[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:04 dist-test-slave-f864 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:04 dist-test-slave-f864 krb5kdc[399](info): set up 2 sockets
Sep 26 13:14:04 dist-test-slave-f864 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 13:14:05.354664 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:05.482102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45545
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:05.562134 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:05.562423 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:05.562448 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:05.563544 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:05.563578 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:05.563592 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:05.563606 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:05.563617 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:05.565001 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:05.565189 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:05.565371 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:05.570538 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:05.570538 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:05.570802 412 server_base.cc:1034] running on GCE node
W20260926 13:14:05.570672 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:05.571154 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:05.571424 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:05.572569 412 hybrid_clock.cc:648] HybridClock initialized: now 1790428445572554 us; error 33 us; skew 500 ppm
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:05.575065 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:05.575544 412 webserver.cc:460] Webserver started at http://127.0.98.62:44149/ using document root <none> and password file <none>
I20260926 13:14:05.575791 412 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:05.575860 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:05.575999 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:05.577111 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "ba85d1b899a544e1acc6ba32c867485d"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "62c69d9bfc736fa5c89b01dcbbd09fb2"
server_key_iv: "c55939c08e852b635eff385ed6e6b115"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.577504 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba85d1b899a544e1acc6ba32c867485d"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "62c69d9bfc736fa5c89b01dcbbd09fb2"
server_key_iv: "c55939c08e852b635eff385ed6e6b115"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.578950 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:14:05.579684 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.579856 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:05.579950 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "ba85d1b899a544e1acc6ba32c867485d"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "62c69d9bfc736fa5c89b01dcbbd09fb2"
server_key_iv: "c55939c08e852b635eff385ed6e6b115"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.580031 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:05.592713 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:05.594820 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:05.594997 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:05.599129 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45545
I20260926 13:14:05.599263 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45545 every 8 connection(s)
I20260926 13:14:05.599588 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:05.600502 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:05.602898 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d: Bootstrap starting.
I20260926 13:14:05.603622 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:05.603929 480 log.cc:824] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:05.604652 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d: No bootstrap required, opened a new log
I20260926 13:14:05.606333 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba85d1b899a544e1acc6ba32c867485d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45545 } }
I20260926 13:14:05.606468 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:05.606535 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba85d1b899a544e1acc6ba32c867485d, State: Initialized, Role: FOLLOWER
I20260926 13:14:05.606694 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba85d1b899a544e1acc6ba32c867485d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45545 } }
I20260926 13:14:05.606777 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:05.606810 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:05.606863 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:05.606837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 412
I20260926 13:14:05.606935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:05.607610 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba85d1b899a544e1acc6ba32c867485d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45545 } }
I20260926 13:14:05.607740 392 external_mini_cluster.cc:1428] Setting key 48ecb7b1d659458fe2b12bf691fab598
I20260926 13:14:05.607726 480 leader_election.cc:302] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba85d1b899a544e1acc6ba32c867485d; no voters:
I20260926 13:14:05.607905 480 leader_election.cc:288] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:05.607959 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:05.608119 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [term 1 LEADER]: Becoming Leader. State: Replica: ba85d1b899a544e1acc6ba32c867485d, State: Running, Role: LEADER
I20260926 13:14:05.608162 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:05.608214 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba85d1b899a544e1acc6ba32c867485d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45545 } }
I20260926 13:14:05.608505 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba85d1b899a544e1acc6ba32c867485d. Latest consensus state: current_term: 1 leader_uuid: "ba85d1b899a544e1acc6ba32c867485d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba85d1b899a544e1acc6ba32c867485d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45545 } } }
I20260926 13:14:05.608585 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:05.608764 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba85d1b899a544e1acc6ba32c867485d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba85d1b899a544e1acc6ba32c867485d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45545 } } }
I20260926 13:14:05.608888 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d [sys.catalog]: This master's current role is: LEADER
W20260926 13:14:05.610234 501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:05.610292 501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:05.610345 489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:05.610493 489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:05.612097 489 catalog_manager.cc:1347] Generated new cluster ID: f0745c8e1ad94ec2ad6d2fa318b7f1be
I20260926 13:14:05.612179 489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:05.623374 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:05.609285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.0.98.62:45545)
0926 13:14:05.609492 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:05.609501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:05.609802 (+ 301us) server_negotiation.cc:407] Connection header received
0926 13:14:05.612115 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:05.612121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:05.612180 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:05.612324 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:05.613791 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.614820 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.615845 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.616253 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.616389 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:05.619798 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:05.619801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:05.619802 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:05.619811 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:05.621468 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:05.621906 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:05.621908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:05.621909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:05.621995 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:05.622337 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:05.622338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:05.622339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:05.622568 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:05.622745 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:05.623023 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 13:14:05.623156 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260926 13:14:05.634693 489 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:05.635447 489 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:05.650363 489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba85d1b899a544e1acc6ba32c867485d: Generated new TSK 0
I20260926 13:14:05.650610 489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:05.678730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45545
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:05.761514 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:05.761780 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:05.761824 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:05.763041 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:05.763114 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:05.763160 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:05.764621 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:05.764901 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:05.765136 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:05.770454 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:05.770478 506 server_base.cc:1034] running on GCE node
W20260926 13:14:05.770421 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:05.770440 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:05.770872 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:05.771148 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:05.772310 506 hybrid_clock.cc:648] HybridClock initialized: now 1790428445772300 us; error 35 us; skew 500 ppm
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:05.774935 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:05.775369 506 webserver.cc:460] Webserver started at http://127.0.98.1:39885/ using document root <none> and password file <none>
I20260926 13:14:05.775592 506 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:05.775660 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:05.775821 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:05.776856 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "35ecdc5b50764e4e8442977f9b086aba"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "301e306492fe33a6d1f7b38bbf4be037"
server_key_iv: "0b67b094b3f2fa395afbf1b3be822ef4"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.777215 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "35ecdc5b50764e4e8442977f9b086aba"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "301e306492fe33a6d1f7b38bbf4be037"
server_key_iv: "0b67b094b3f2fa395afbf1b3be822ef4"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.778582 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:14:05.779382 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.779557 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:05.779659 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "35ecdc5b50764e4e8442977f9b086aba"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "301e306492fe33a6d1f7b38bbf4be037"
server_key_iv: "0b67b094b3f2fa395afbf1b3be822ef4"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.779729 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:05.793987 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:05.796031 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:05.796200 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:05.796445 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:05.796746 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:05.796799 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.796835 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:05.796877 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.802543 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42075
I20260926 13:14:05.802600 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42075 every 8 connection(s)
I20260926 13:14:05.802959 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:14:05.803928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 506
I20260926 13:14:05.804021 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:05.804188 392 external_mini_cluster.cc:1428] Setting key 1a341a4eb8d4198cfbdd99a19561ca1d
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:14:05.814128 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:05.803699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43821 (local address 127.0.98.62:45545)
0926 13:14:05.803781 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:05.803785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:05.804065 (+ 280us) server_negotiation.cc:407] Connection header received
0926 13:14:05.804753 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:05.804754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:05.804767 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:05.804827 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:05.806141 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.806688 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.807813 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.808520 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.808596 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:05.811041 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:05.811042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:05.811042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:05.811052 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:05.812706 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:05.813092 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:05.813093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:05.813094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:05.813159 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:05.813426 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:05.813427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:05.813428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:05.813574 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:05.813652 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:05.813907 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 13:14:05.814012 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:14:05.814390 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:05.803668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45545 (local address 127.0.98.1:43821)
0926 13:14:05.803924 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:05.803971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:05.804647 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:05.804909 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:05.804913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:05.805759 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:05.806044 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:05.806050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.806796 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.806798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:05.807706 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:05.807709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.808604 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.808605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:05.808776 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:05.809436 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:05.809444 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:05.810944 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:05.812784 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:05.812790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:05.812797 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:05.813012 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:05.813245 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:05.813246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:05.813246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:05.813351 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:05.813646 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:05.813651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:05.813766 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 13:14:05.813855 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:14:05.814050 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
I20260926 13:14:05.814952 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45545
I20260926 13:14:05.815039 635 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:05.815232 635 heartbeater.cc:511] Master 127.0.98.62:45545 requested a full tablet report, sending...
I20260926 13:14:05.815776 444 ts_manager.cc:194] Registered new tserver with Master: 35ecdc5b50764e4e8442977f9b086aba (127.0.98.1:42075)
I20260926 13:14:05.816447 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:05.832965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45545
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:05.914151 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:05.914389 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:05.914415 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:05.915617 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:05.915656 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:05.915683 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:05.917171 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:05.917441 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:05.917685 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:05.922896 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:05.923053 640 server_base.cc:1034] running on GCE node
W20260926 13:14:05.923149 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:05.922923 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:05.923425 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:05.923686 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:05.924853 640 hybrid_clock.cc:648] HybridClock initialized: now 1790428445924837 us; error 34 us; skew 500 ppm
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:05.927561 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:05.928122 640 webserver.cc:460] Webserver started at http://127.0.98.2:39361/ using document root <none> and password file <none>
I20260926 13:14:05.928371 640 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:05.928442 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:05.928596 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:05.929692 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "b4c72c8ccfb2424391f17f94dbcc9f3a"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "9229d85d7b9d00851cc32990f6d7e36c"
server_key_iv: "fdb083bebff163771551ad8ea11559a9"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.930090 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b4c72c8ccfb2424391f17f94dbcc9f3a"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "9229d85d7b9d00851cc32990f6d7e36c"
server_key_iv: "fdb083bebff163771551ad8ea11559a9"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.931504 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:05.932329 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.932528 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:05.932636 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "b4c72c8ccfb2424391f17f94dbcc9f3a"
format_stamp: "Formatted at 2026-09-26 13:14:05 on dist-test-slave-f864"
server_key: "9229d85d7b9d00851cc32990f6d7e36c"
server_key_iv: "fdb083bebff163771551ad8ea11559a9"
server_key_version: "encryptionkey@0"
I20260926 13:14:05.932710 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:05.939828 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:05.942370 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:05.942539 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:05.942785 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:05.943116 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:05.943169 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.943210 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:05.943256 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:05.949536 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35833
I20260926 13:14:05.949584 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35833 every 8 connection(s)
I20260926 13:14:05.949910 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:14:05 dist-test-slave-f864 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:05.957916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 640
I20260926 13:14:05.958003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:05.958246 392 external_mini_cluster.cc:1428] Setting key b803f27751b72aaf36e903badcfdc946
I20260926 13:14:05.959789 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:05.950682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57243 (local address 127.0.98.62:45545)
0926 13:14:05.950745 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:05.950747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:05.950950 (+ 203us) server_negotiation.cc:407] Connection header received
0926 13:14:05.951431 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:05.951432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:05.951441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:05.951501 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:05.952442 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.952952 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.953955 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.954429 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.954499 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:05.956716 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:05.956718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:05.956718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:05.956726 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:05.958024 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:05.958634 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:05.958635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:05.958635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:05.958701 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:05.959044 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:05.959045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:05.959046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:05.959195 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:05.959292 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:05.959585 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 13:14:05.959681 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:14:05.959890 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:05.950660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45545 (local address 127.0.98.2:57243)
0926 13:14:05.950814 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:05.950860 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:05.951347 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:05.951555 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:05.951558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:05.952128 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:05.952331 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:05.952336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.953032 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.953032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:05.953866 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:05.953869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:05.954534 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:05.954535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:05.954655 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:05.955254 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:05.955263 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:05.956617 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:05.958131 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:05.958137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:05.958139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:05.958542 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:05.958790 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:05.958791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:05.958791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:05.958956 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:05.959304 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:05.959310 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:05.959443 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 13:14:05.959543 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:14:05.959675 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260926 13:14:05.960371 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45545
I20260926 13:14:05.960490 769 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:05.960697 769 heartbeater.cc:511] Master 127.0.98.62:45545 requested a full tablet report, sending...
I20260926 13:14:05.961134 444 ts_manager.cc:194] Registered new tserver with Master: b4c72c8ccfb2424391f17f94dbcc9f3a (127.0.98.2:35833)
I20260926 13:14:05.961581 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57243
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:05.987232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45545
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:06.069159 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:06.069371 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:06.069396 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:06.070510 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:06.070546 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:06.070572 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:06.071947 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:06.072147 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:06.072317 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:14:06.077481 774 server_base.cc:1034] running on GCE node
W20260926 13:14:06.077489 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:06.077463 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:06.077454 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:06.077970 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:06.078290 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:06.079470 774 hybrid_clock.cc:648] HybridClock initialized: now 1790428446079460 us; error 22 us; skew 500 ppm
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:06.082455 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:06.082984 774 webserver.cc:460] Webserver started at http://127.0.98.3:33373/ using document root <none> and password file <none>
I20260926 13:14:06.083217 774 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:06.083284 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:06.083429 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:06.084511 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "939ec19a854b4c95a07d006a40c355fc"
format_stamp: "Formatted at 2026-09-26 13:14:06 on dist-test-slave-f864"
server_key: "96eb30a806aa14942f49a631be62cb00"
server_key_iv: "e3592653443f117e9456386d0ff8ff18"
server_key_version: "encryptionkey@0"
I20260926 13:14:06.084884 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "939ec19a854b4c95a07d006a40c355fc"
format_stamp: "Formatted at 2026-09-26 13:14:06 on dist-test-slave-f864"
server_key: "96eb30a806aa14942f49a631be62cb00"
server_key_iv: "e3592653443f117e9456386d0ff8ff18"
server_key_version: "encryptionkey@0"
I20260926 13:14:06.086330 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:06.087074 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:06.087267 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:06.087366 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "939ec19a854b4c95a07d006a40c355fc"
format_stamp: "Formatted at 2026-09-26 13:14:06 on dist-test-slave-f864"
server_key: "96eb30a806aa14942f49a631be62cb00"
server_key_iv: "e3592653443f117e9456386d0ff8ff18"
server_key_version: "encryptionkey@0"
I20260926 13:14:06.087455 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:06.104049 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:06.106570 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:06.106681 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:06.106870 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:06.107172 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:06.107203 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:06.107226 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:06.107273 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:06.113956 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35281
I20260926 13:14:06.114044 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35281 every 8 connection(s)
I20260926 13:14:06.114763 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:06.122054 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 774
I20260926 13:14:06.122154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:06.122359 392 external_mini_cluster.cc:1428] Setting key bcc11a822c803ebe05638c1b9448e12a
I20260926 13:14:06.124645 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:06.115516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60609 (local address 127.0.98.62:45545)
0926 13:14:06.115563 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:06.115565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:06.115788 (+ 223us) server_negotiation.cc:407] Connection header received
0926 13:14:06.116350 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:06.116352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:06.116366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:06.116423 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:06.117486 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.118063 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.119293 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.119496 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.119554 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:06.121688 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:06.121689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:06.121689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:06.121695 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:06.123504 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:06.123937 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:06.123938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:06.123939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:06.123990 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:06.124175 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:06.124175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:06.124176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:06.124274 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:06.124325 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:06.124524 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:14:06.124586 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:14:06.124820 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:06.115495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45545 (local address 127.0.98.3:60609)
0926 13:14:06.115638 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:06.115673 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:06.116231 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:06.116501 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:06.116505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:06.117160 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:06.117369 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:06.117373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.118193 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.118194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:06.119182 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:06.119185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.119590 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.119591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:06.119698 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:06.120279 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:06.120287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:06.121606 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:06.123587 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:06.123593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:06.123596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:06.123867 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:06.124037 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:06.124038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:06.124038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:06.124124 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:06.124312 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:06.124315 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:06.124421 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 13:14:06.124497 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:14:06.124615 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":98,"threads_started":1}
I20260926 13:14:06.125329 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45545
I20260926 13:14:06.125432 903 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:06.125597 903 heartbeater.cc:511] Master 127.0.98.62:45545 requested a full tablet report, sending...
I20260926 13:14:06.125983 444 ts_manager.cc:194] Registered new tserver with Master: 939ec19a854b4c95a07d006a40c355fc (127.0.98.3:35281)
I20260926 13:14:06.126420 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60609
I20260926 13:14:06.134361 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428446, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428446, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:06.157559 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:06.149532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.98.1:42075)
0926 13:14:06.149727 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:06.149731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:06.149742 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:06.149768 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:06.149770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:06.149784 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:06.149852 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:06.150584 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.151328 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.152103 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.152439 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.152536 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:06.154935 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:06.154941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:06.154942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:06.154949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:06.156437 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:06.156787 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:06.156790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:06.156790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:06.156868 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:06.157045 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:06.157045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:06.157046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:06.157172 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:06.157275 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:06.157417 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:14:06.157480 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
W20260926 13:14:06.157948 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47752
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:06.180359 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:06.172849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.0.98.1:42075)
0926 13:14:06.172940 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:06.172942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:06.173026 (+ 84us) server_negotiation.cc:407] Connection header received
0926 13:14:06.173127 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:06.173128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:06.173140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:06.173221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:06.173787 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.174369 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.175124 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:06.175344 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:06.175417 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:06.177765 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:06.177767 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:06.177767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:06.177776 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:06.179239 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:06.179599 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:06.179600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:06.179601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:06.179658 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:06.179857 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:06.179858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:06.179858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:06.179957 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:06.180083 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:06.180169 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:14:06.180254 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 13:14:06.181001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 506
I20260926 13:14:06.186305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 640
I20260926 13:14:06.192587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 774
I20260926 13:14:06.197269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 412
2026-09-26T13:14:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1911 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[917](info): set up 2 sockets
Sep 26 13:14:06 dist-test-slave-f864 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 13:14:07.245762 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:07.363271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:07.444528 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:07.444796 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:07.444834 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:07.446029 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:07.446110 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:07.446134 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:07.446156 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:07.446173 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:07.447726 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:07.448024 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:07.448264 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:07.453222 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.453222 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.453276 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:07.453547 930 server_base.cc:1034] running on GCE node
I20260926 13:14:07.453747 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:07.454011 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:07.455163 930 hybrid_clock.cc:648] HybridClock initialized: now 1790428447455133 us; error 43 us; skew 500 ppm
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:07.457767 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:07.458328 930 webserver.cc:460] Webserver started at http://127.0.98.62:44523/ using document root <none> and password file <none>
I20260926 13:14:07.458546 930 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:07.458590 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:07.458685 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:07.459754 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "313fd77fb800423db046722d20f1925c"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "6db2e12fdddcb9e7c7e9d4960dec96a7"
server_key_iv: "a436c15e8ea9a718c850e338c737ec85"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.460069 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "313fd77fb800423db046722d20f1925c"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "6db2e12fdddcb9e7c7e9d4960dec96a7"
server_key_iv: "a436c15e8ea9a718c850e338c737ec85"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.461416 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:07.462260 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.462461 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:07.462566 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "313fd77fb800423db046722d20f1925c"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "6db2e12fdddcb9e7c7e9d4960dec96a7"
server_key_iv: "a436c15e8ea9a718c850e338c737ec85"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.462658 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:07.471764 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:07.473661 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:07.473834 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:07.478232 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35323
I20260926 13:14:07.478257 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35323 every 8 connection(s)
I20260926 13:14:07.478672 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:07.479596 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:07.481930 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c: Bootstrap starting.
I20260926 13:14:07.482583 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:07.482882 998 log.cc:824] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:07.483566 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c: No bootstrap required, opened a new log
I20260926 13:14:07.485121 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313fd77fb800423db046722d20f1925c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35323 } }
I20260926 13:14:07.485285 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:07.485332 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313fd77fb800423db046722d20f1925c, State: Initialized, Role: FOLLOWER
I20260926 13:14:07.485535 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313fd77fb800423db046722d20f1925c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35323 } }
I20260926 13:14:07.485647 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:07.485693 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:07.485745 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:07.486296 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313fd77fb800423db046722d20f1925c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35323 } }
I20260926 13:14:07.486414 998 leader_election.cc:302] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313fd77fb800423db046722d20f1925c; no voters:
I20260926 13:14:07.486613 998 leader_election.cc:288] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:07.486797 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:07.486873 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:07.487190 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [term 1 LEADER]: Becoming Leader. State: Replica: 313fd77fb800423db046722d20f1925c, State: Running, Role: LEADER
I20260926 13:14:07.487283 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313fd77fb800423db046722d20f1925c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35323 } }
I20260926 13:14:07.487663 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313fd77fb800423db046722d20f1925c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313fd77fb800423db046722d20f1925c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35323 } } }
I20260926 13:14:07.487677 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313fd77fb800423db046722d20f1925c. Latest consensus state: current_term: 1 leader_uuid: "313fd77fb800423db046722d20f1925c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313fd77fb800423db046722d20f1925c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35323 } } }
I20260926 13:14:07.487746 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:07.487777 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:07.487931 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:07.488333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 930
I20260926 13:14:07.488430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:07.488588 392 external_mini_cluster.cc:1428] Setting key 4798cb05f7f693cdedc3febc27c6bc8d
I20260926 13:14:07.489156 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:07.490854 1011 catalog_manager.cc:1347] Generated new cluster ID: a9dfc59655a44fc58d2daed81a48b195
I20260926 13:14:07.490906 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:07.500946 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:07.489301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.0.98.62:35323)
0926 13:14:07.489459 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:07.489470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:07.489661 (+ 191us) server_negotiation.cc:407] Connection header received
0926 13:14:07.489782 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:07.489790 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:07.489851 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:07.490023 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:07.490810 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.492215 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.493295 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.493923 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.494062 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:07.496931 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:07.496936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:07.496936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:07.496950 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:07.499094 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:07.499522 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:07.499525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:07.499526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:07.499649 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:07.499860 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:07.499861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:07.499861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:07.500182 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:07.500318 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:07.500458 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 13:14:07.500694 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260926 13:14:07.513902 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:07.514685 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:07.529950 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313fd77fb800423db046722d20f1925c: Generated new TSK 0
I20260926 13:14:07.530224 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:07.563508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35323
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:07.654139 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:07.654399 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:07.654428 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:07.655704 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:07.655746 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:07.655776 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:07.657298 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:07.657510 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:07.657714 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:07.663236 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.663280 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.663252 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:07.663542 1024 server_base.cc:1034] running on GCE node
I20260926 13:14:07.663786 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:07.664003 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:07.665231 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790428447665184 us; error 30 us; skew 500 ppm
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:07.667985 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:07.668533 1024 webserver.cc:460] Webserver started at http://127.0.98.1:42621/ using document root <none> and password file <none>
I20260926 13:14:07.668747 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:07.668788 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:07.668885 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:07.669919 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "aba56fc9988e4f27975e5aa1c08f4f19"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "2411bb31b7ff4eed13b5d9e89de42c03"
server_key_iv: "3bf77f14c25cfd91b817f98d491723f9"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.670292 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aba56fc9988e4f27975e5aa1c08f4f19"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "2411bb31b7ff4eed13b5d9e89de42c03"
server_key_iv: "3bf77f14c25cfd91b817f98d491723f9"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.671599 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:07.672334 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.672557 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:07.672663 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "aba56fc9988e4f27975e5aa1c08f4f19"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "2411bb31b7ff4eed13b5d9e89de42c03"
server_key_iv: "3bf77f14c25cfd91b817f98d491723f9"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.672735 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:07.703500 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:07.705435 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:07.705590 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:07.705859 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:07.706208 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:07.706243 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.706266 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:07.706312 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.712352 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37777
I20260926 13:14:07.712432 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37777 every 8 connection(s)
I20260926 13:14:07.712782 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:14:07.719223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1024
I20260926 13:14:07.719300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:07.719507 392 external_mini_cluster.cc:1428] Setting key 0e3b911b9dd564c7399ff3c2b7ce0629
I20260926 13:14:07.723099 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:07.713570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35323 (local address 127.0.98.1:35307)
0926 13:14:07.713725 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:07.713761 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:07.714303 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:07.714531 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:07.714535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:07.715114 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:07.715329 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:07.715335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.716062 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.716063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:07.716978 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:07.716981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.717418 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.717419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:07.717533 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:07.718125 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:07.718136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:07.719865 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:07.721669 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:07.721676 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:07.721679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:07.721911 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:07.722146 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:07.722147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:07.722148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:07.722295 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:07.722576 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:07.722582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:07.722711 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:14:07.722798 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:14:07.722927 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260926 13:14:07.723096 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:07.713596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35307 (local address 127.0.98.62:35323)
0926 13:14:07.713639 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:07.713641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:07.713856 (+ 215us) server_negotiation.cc:407] Connection header received
0926 13:14:07.714397 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:07.714398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:07.714412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:07.714466 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:07.715432 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.715933 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.717096 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.717311 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.717384 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:07.719963 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:07.719965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:07.719966 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:07.719974 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:07.721562 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:07.722008 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:07.722010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:07.722010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:07.722071 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:07.722365 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:07.722366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:07.722366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:07.722508 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:07.722585 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:07.722860 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 13:14:07.722955 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:14:07.723912 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35323
I20260926 13:14:07.724030 1153 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:07.724241 1153 heartbeater.cc:511] Master 127.0.98.62:35323 requested a full tablet report, sending...
I20260926 13:14:07.724951 962 ts_manager.cc:194] Registered new tserver with Master: aba56fc9988e4f27975e5aa1c08f4f19 (127.0.98.1:37777)
I20260926 13:14:07.725551 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35307
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:07.745879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35323
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:07.828018 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:07.828219 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:07.828243 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:07.829327 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:07.829362 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:07.829388 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:07.830788 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:07.830964 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:07.831130 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:07.835912 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.835919 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.835976 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:07.836150 1158 server_base.cc:1034] running on GCE node
I20260926 13:14:07.836350 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:07.836651 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:07.837826 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790428447837814 us; error 25 us; skew 500 ppm
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:07.840467 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:07.840942 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43139/ using document root <none> and password file <none>
I20260926 13:14:07.841173 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:07.841239 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:07.841391 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:07.842551 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "f95d1fc4497d4980b151a6055f561d8c"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "d45b882334c76db929bab0789de9704e"
server_key_iv: "42101861d0662dd5fb50c553b15d5ca4"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.842902 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f95d1fc4497d4980b151a6055f561d8c"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "d45b882334c76db929bab0789de9704e"
server_key_iv: "42101861d0662dd5fb50c553b15d5ca4"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.844199 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:07.844935 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.845108 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:07.845216 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "f95d1fc4497d4980b151a6055f561d8c"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "d45b882334c76db929bab0789de9704e"
server_key_iv: "42101861d0662dd5fb50c553b15d5ca4"
server_key_version: "encryptionkey@0"
I20260926 13:14:07.845293 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:07.857622 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:07.859618 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:07.859771 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:07.859993 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:07.860291 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:07.860345 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.860387 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:07.860430 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:07.866714 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40851
I20260926 13:14:07.866755 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40851 every 8 connection(s)
I20260926 13:14:07.867079 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:07.870769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1158
I20260926 13:14:07.870843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:07.871018 392 external_mini_cluster.cc:1428] Setting key fe71a2091eed479303909a52b7c35a64
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:07.876612 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:07.867737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49575 (local address 127.0.98.62:35323)
0926 13:14:07.867808 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:07.867810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:07.868018 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:14:07.868473 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:07.868474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:07.868484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:07.868549 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:07.869424 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.869904 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.871009 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.871302 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.871371 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:07.873697 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:07.873699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:07.873699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:07.873706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:07.875179 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:07.875642 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:07.875643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:07.875644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:07.875707 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:07.875957 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:07.875958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:07.875958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:07.876074 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:07.876148 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:07.876409 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 13:14:07.876507 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:14:07.876808 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:07.867726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35323 (local address 127.0.98.2:49575)
0926 13:14:07.867880 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:07.867922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:07.868398 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:07.868604 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:07.868607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:07.869162 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:07.869346 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:07.869351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.869978 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.869979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:07.870922 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:07.870925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:07.871402 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:07.871403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:07.871556 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:07.872102 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:07.872111 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:07.873601 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:07.875269 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:07.875275 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:07.875277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:07.875567 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:07.875775 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:07.875776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:07.875777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:07.875889 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:07.876140 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:07.876146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:07.876282 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 13:14:07.876390 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 13:14:07.876539 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260926 13:14:07.877363 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35323
I20260926 13:14:07.877468 1287 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:07.877638 1287 heartbeater.cc:511] Master 127.0.98.62:35323 requested a full tablet report, sending...
I20260926 13:14:07.878033 962 ts_manager.cc:194] Registered new tserver with Master: f95d1fc4497d4980b151a6055f561d8c (127.0.98.2:40851)
I20260926 13:14:07.878592 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49575
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:07.896510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35323
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:07.977676 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:07.977952 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:07.978005 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:07.979307 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:07.979395 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:07.979441 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:07.981060 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35323
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:07.981364 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:07.981616 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:07.986825 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.986889 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:07.986871 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:07.987020 1292 server_base.cc:1034] running on GCE node
I20260926 13:14:07.987251 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:07.987511 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:07.988703 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790428447988686 us; error 27 us; skew 500 ppm
Sep 26 13:14:07 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:07.999133 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:07.999683 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36181/ using document root <none> and password file <none>
I20260926 13:14:07.999925 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:08.000003 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:08.000161 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:08.001416 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "6ad5f2319aa9483b95a80bfbe1f107ff"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "145dd152cc89e6c26693ea455d330585"
server_key_iv: "6701123747ac897bee35e16a6bbf84a8"
server_key_version: "encryptionkey@0"
I20260926 13:14:08.001791 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6ad5f2319aa9483b95a80bfbe1f107ff"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "145dd152cc89e6c26693ea455d330585"
server_key_iv: "6701123747ac897bee35e16a6bbf84a8"
server_key_version: "encryptionkey@0"
I20260926 13:14:08.003356 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:14:08.004318 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:08.004522 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:08.004638 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "6ad5f2319aa9483b95a80bfbe1f107ff"
format_stamp: "Formatted at 2026-09-26 13:14:07 on dist-test-slave-f864"
server_key: "145dd152cc89e6c26693ea455d330585"
server_key_iv: "6701123747ac897bee35e16a6bbf84a8"
server_key_version: "encryptionkey@0"
I20260926 13:14:08.004734 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:08.044337 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:08.047163 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:08.047353 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:08.047636 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:08.048058 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:08.048116 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:08.048163 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:08.048194 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:08.054968 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40601
I20260926 13:14:08.055009 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40601 every 8 connection(s)
I20260926 13:14:08.055415 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:08.061664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1292
I20260926 13:14:08.061749 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:08.061911 392 external_mini_cluster.cc:1428] Setting key 3e77fb78e6a3cce84cb9c06f77192faf
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:08.065549 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.056218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34037 (local address 127.0.98.62:35323)
0926 13:14:08.056286 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.056292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.056509 (+ 217us) server_negotiation.cc:407] Connection header received
0926 13:14:08.057041 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.057042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.057055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.057097 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:08.058107 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.058680 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.059722 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.059923 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.060012 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.062587 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:08.062589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:08.062589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:08.062599 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:08.064231 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.064616 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.064617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.064617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.064674 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.064889 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.064890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.064890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.064998 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:08.065071 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.065398 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.065479 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:14:08.065699 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.056214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35323 (local address 127.0.98.3:34037)
0926 13:14:08.056383 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:08.056420 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:08.056931 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:08.057189 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:08.057192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:08.057787 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:08.057991 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.057996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.058784 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.058785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.059625 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.059627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.060024 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.060025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.060156 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.060702 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:08.060711 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:08.062480 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:08.064308 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:08.064314 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:08.064315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:08.064529 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:08.064740 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:08.064741 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:08.064741 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:08.064835 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:08.065087 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:08.065091 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:08.065248 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 13:14:08.065368 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 13:14:08.065489 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260926 13:14:08.066180 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35323
I20260926 13:14:08.066277 1421 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:08.066463 1421 heartbeater.cc:511] Master 127.0.98.62:35323 requested a full tablet report, sending...
I20260926 13:14:08.066831 962 ts_manager.cc:194] Registered new tserver with Master: 6ad5f2319aa9483b95a80bfbe1f107ff (127.0.98.3:40601)
I20260926 13:14:08.067159 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34037
I20260926 13:14:08.073837 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:14:08.081440 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.075487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.0.98.62:35323)
0926 13:14:08.075563 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.075565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.075661 (+ 96us) server_negotiation.cc:407] Connection header received
0926 13:14:08.075703 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.075704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.075712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.075763 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:08.076362 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.076847 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.077524 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.077710 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.077759 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.079103 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:08.079104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:08.079104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:08.079110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:08.080372 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.080715 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.080716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.080716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.080772 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.080961 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.080961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.080962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.081117 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:08.081202 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.081302 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.081372 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:14:08.083719 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:08.085239 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:08.091728 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.087886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59476 (local address 127.0.98.2:40851)
0926 13:14:08.088255 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.088259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.088268 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:08.088302 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.088304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.088318 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.088383 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:08.088599 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.089326 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.090777 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.091342 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.091429 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.091490 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.091605 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.091644 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":76,"threads_started":1}
I20260926 13:14:08.091717 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.087664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40851 (local address 127.0.0.1:59476)
0926 13:14:08.087986 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:08.088041 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:08.088113 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:08.088456 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:08.088457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:08.088459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:08.088521 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.088523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.089429 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.089431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.090696 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.090698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.091428 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.091429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.091502 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.091516 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:14:08.091590 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:14:08.091641 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":6,"thread_start_us":79,"threads_started":1}
I20260926 13:14:08.092211 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.087867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40601 (local address 127.0.0.1:44918)
0926 13:14:08.087983 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:08.088023 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:08.088136 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:08.088463 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:08.088464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:08.088466 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:08.088513 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.088514 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.089812 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.089813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.090881 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.090883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.091972 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.091973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.092037 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.092041 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:08.092110 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:14:08.092148 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":52,"thread_start_us":72,"threads_started":1}
I20260926 13:14:08.092415 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.087664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37777 (local address 127.0.0.1:37078)
0926 13:14:08.087838 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:08.088127 (+ 289us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:08.088179 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:08.088351 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:08.088354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:08.088356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:08.088430 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.088432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.089808 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.089809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.090907 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.090909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.092061 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.092062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.092139 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.092142 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:08.092185 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:14:08.092225 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":31,"thread_start_us":62,"threads_started":1}
I20260926 13:14:08.092432 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.087780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37078 (local address 127.0.98.1:37777)
0926 13:14:08.088112 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.088118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.088196 (+ 78us) server_negotiation.cc:407] Connection header received
0926 13:14:08.088242 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.088245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.088262 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.088345 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:08.088518 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.089708 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.091022 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.091730 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.091802 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.092284 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.092313 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.092363 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":91,"threads_started":1}
I20260926 13:14:08.092461 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0993654ad76b41b094b6d3b555151adc (DEFAULT_TABLE table=test-table [id=bc549de07f0643e8b20c76c1180dcd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:08.092254 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.087958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44918 (local address 127.0.98.3:40601)
0926 13:14:08.088253 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.088258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.088266 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:14:08.088331 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.088333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.088349 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.088587 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:08.088642 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.089689 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.090966 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.091729 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.091971 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.092027 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.092125 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.092173 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":79,"threads_started":1}
I20260926 13:14:08.092908 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0993654ad76b41b094b6d3b555151adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:08.094811 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 0993654ad76b41b094b6d3b555151adc (DEFAULT_TABLE table=test-table [id=bc549de07f0643e8b20c76c1180dcd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:08.095134 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0993654ad76b41b094b6d3b555151adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:08.095060 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0993654ad76b41b094b6d3b555151adc (DEFAULT_TABLE table=test-table [id=bc549de07f0643e8b20c76c1180dcd09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:08.095362 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0993654ad76b41b094b6d3b555151adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:08.095762 1438 tablet_bootstrap.cc:492] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: Bootstrap starting.
I20260926 13:14:08.096557 1438 tablet_bootstrap.cc:654] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:08.097177 1438 log.cc:824] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:08.098011 1438 tablet_bootstrap.cc:492] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: No bootstrap required, opened a new log
I20260926 13:14:08.098104 1438 ts_tablet_manager.cc:1397] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:08.098798 1441 tablet_bootstrap.cc:492] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: Bootstrap starting.
I20260926 13:14:08.099534 1441 tablet_bootstrap.cc:654] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:08.100172 1441 log.cc:824] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:08.100564 1440 tablet_bootstrap.cc:492] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: Bootstrap starting.
I20260926 13:14:08.100597 1438 raft_consensus.cc:348] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.100785 1438 raft_consensus.cc:374] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:08.100824 1438 raft_consensus.cc:729] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95d1fc4497d4980b151a6055f561d8c, State: Initialized, Role: FOLLOWER
I20260926 13:14:08.101006 1438 consensus_queue.cc:260] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.101059 1441 tablet_bootstrap.cc:492] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: No bootstrap required, opened a new log
I20260926 13:14:08.101150 1441 ts_tablet_manager.cc:1397] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:08.101289 1440 tablet_bootstrap.cc:654] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:08.101300 1438 ts_tablet_manager.cc:1428] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:14:08.101421 1287 heartbeater.cc:503] Master 127.0.98.62:35323 was elected leader, sending a full tablet report...
I20260926 13:14:08.102210 1440 log.cc:824] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:08.103047 1440 tablet_bootstrap.cc:492] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: No bootstrap required, opened a new log
I20260926 13:14:08.103099 1440 ts_tablet_manager.cc:1397] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:14:08.103785 1441 raft_consensus.cc:348] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.103953 1441 raft_consensus.cc:374] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:08.103999 1441 raft_consensus.cc:729] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aba56fc9988e4f27975e5aa1c08f4f19, State: Initialized, Role: FOLLOWER
I20260926 13:14:08.104130 1441 consensus_queue.cc:260] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.104450 1441 ts_tablet_manager.cc:1428] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 13:14:08.104532 1153 heartbeater.cc:503] Master 127.0.98.62:35323 was elected leader, sending a full tablet report...
I20260926 13:14:08.104636 1440 raft_consensus.cc:348] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.104760 1440 raft_consensus.cc:374] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:08.104818 1440 raft_consensus.cc:729] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ad5f2319aa9483b95a80bfbe1f107ff, State: Initialized, Role: FOLLOWER
I20260926 13:14:08.104943 1440 consensus_queue.cc:260] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.105245 1440 ts_tablet_manager.cc:1428] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:08.105247 1421 heartbeater.cc:503] Master 127.0.98.62:35323 was elected leader, sending a full tablet report...
W20260926 13:14:08.117599 1288 tablet.cc:2369] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:14:08.213577 1154 tablet.cc:2369] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:14:08.260164 1443 raft_consensus.cc:482] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:08.260284 1443 raft_consensus.cc:504] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.260648 1443 leader_election.cc:288] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aba56fc9988e4f27975e5aa1c08f4f19 (127.0.98.1:37777), 6ad5f2319aa9483b95a80bfbe1f107ff (127.0.98.3:40601)
I20260926 13:14:08.264722 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.260902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43811 (local address 127.0.98.1:37777)
0926 13:14:08.260965 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.260967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.260981 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:14:08.261014 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.261015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.261028 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.261096 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:08.261318 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.262207 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.263628 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.264259 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.264344 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.264355 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.264581 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.264623 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:14:08.264722 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.260941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55731 (local address 127.0.98.3:40601)
0926 13:14:08.260985 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.260987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.261150 (+ 163us) server_negotiation.cc:407] Connection header received
0926 13:14:08.261309 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.261310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.261323 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.261408 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:08.261532 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.262358 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.263630 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.264269 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.264344 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.264355 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.264581 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.264622 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:14:08.264791 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.260833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37777 (local address 127.0.98.2:43811)
0926 13:14:08.260894 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:08.260907 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:08.260954 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:08.261135 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:08.261136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:08.261138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:08.261214 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.261217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.262282 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.262283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.263522 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.263524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.264405 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.264406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.264484 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.264486 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:14:08.264563 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:14:08.264632 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":32}
I20260926 13:14:08.264798 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.260859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40601 (local address 127.0.98.2:55731)
0926 13:14:08.261059 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:08.261074 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:08.261157 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:08.261413 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:08.261413 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:08.261414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:08.261459 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.261460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.262424 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.262424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.263521 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:08.263524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.264385 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.264387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:08.264481 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.264486 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:08.264563 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:14:08.264626 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260926 13:14:08.265192 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0993654ad76b41b094b6d3b555151adc" candidate_uuid: "f95d1fc4497d4980b151a6055f561d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" is_pre_election: true
I20260926 13:14:08.265412 1107 raft_consensus.cc:2454] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f95d1fc4497d4980b151a6055f561d8c in term 0.
I20260926 13:14:08.265383 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0993654ad76b41b094b6d3b555151adc" candidate_uuid: "f95d1fc4497d4980b151a6055f561d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" is_pre_election: true
I20260926 13:14:08.265563 1375 raft_consensus.cc:2454] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f95d1fc4497d4980b151a6055f561d8c in term 0.
I20260926 13:14:08.265725 1176 leader_election.cc:302] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aba56fc9988e4f27975e5aa1c08f4f19, f95d1fc4497d4980b151a6055f561d8c; no voters:
I20260926 13:14:08.265880 1443 raft_consensus.cc:2781] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:08.265937 1443 raft_consensus.cc:482] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:08.265961 1443 raft_consensus.cc:3037] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:08.266703 1443 raft_consensus.cc:504] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.266853 1443 leader_election.cc:288] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [CANDIDATE]: Term 1 election: Requested vote from peers aba56fc9988e4f27975e5aa1c08f4f19 (127.0.98.1:37777), 6ad5f2319aa9483b95a80bfbe1f107ff (127.0.98.3:40601)
I20260926 13:14:08.267004 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0993654ad76b41b094b6d3b555151adc" candidate_uuid: "f95d1fc4497d4980b151a6055f561d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aba56fc9988e4f27975e5aa1c08f4f19"
I20260926 13:14:08.267055 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0993654ad76b41b094b6d3b555151adc" candidate_uuid: "f95d1fc4497d4980b151a6055f561d8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff"
I20260926 13:14:08.267096 1107 raft_consensus.cc:3037] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:08.267127 1375 raft_consensus.cc:3037] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:08.268056 1107 raft_consensus.cc:2454] T 0993654ad76b41b094b6d3b555151adc P aba56fc9988e4f27975e5aa1c08f4f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f95d1fc4497d4980b151a6055f561d8c in term 1.
I20260926 13:14:08.268056 1375 raft_consensus.cc:2454] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f95d1fc4497d4980b151a6055f561d8c in term 1.
I20260926 13:14:08.268252 1174 leader_election.cc:302] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ad5f2319aa9483b95a80bfbe1f107ff, f95d1fc4497d4980b151a6055f561d8c; no voters:
I20260926 13:14:08.268375 1443 raft_consensus.cc:2781] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:08.268543 1443 raft_consensus.cc:686] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [term 1 LEADER]: Becoming Leader. State: Replica: f95d1fc4497d4980b151a6055f561d8c, State: Running, Role: LEADER
I20260926 13:14:08.268651 1443 consensus_queue.cc:237] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } }
I20260926 13:14:08.269516 962 catalog_manager.cc:5495] T 0993654ad76b41b094b6d3b555151adc P f95d1fc4497d4980b151a6055f561d8c reported cstate change: term changed from 0 to 1, leader changed from <none> to f95d1fc4497d4980b151a6055f561d8c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f95d1fc4497d4980b151a6055f561d8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aba56fc9988e4f27975e5aa1c08f4f19" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f95d1fc4497d4980b151a6055f561d8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ad5f2319aa9483b95a80bfbe1f107ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40601 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:08.279759 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.272435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.0.98.1:37777)
0926 13:14:08.272506 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.272507 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.272565 (+ 58us) server_negotiation.cc:407] Connection header received
0926 13:14:08.272639 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.272640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.272648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.272693 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:08.273245 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.273709 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.274571 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.274747 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.274798 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.277241 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:08.277250 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:08.277250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:08.277258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:08.278767 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.279080 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.279082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.279082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.279140 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.279302 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.279303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.279303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.279427 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:08.279532 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.279599 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.279657 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428448, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:08.303061 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.295559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.0.98.1:37777)
0926 13:14:08.295654 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.295656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.295749 (+ 93us) server_negotiation.cc:407] Connection header received
0926 13:14:08.295836 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.295837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.295855 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.295922 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:08.296533 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.297045 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.297801 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.298000 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.298080 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.300386 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:08.300389 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:08.300389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:08.300396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:08.301815 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.302194 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.302195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.302195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.302257 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.302458 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.302459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.302460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.302564 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:08.302664 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.302865 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.302944 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 13:14:08.305958 1422 tablet.cc:2369] T 0993654ad76b41b094b6d3b555151adc P 6ad5f2319aa9483b95a80bfbe1f107ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:08.326262 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:08.318409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.98.1:37777)
0926 13:14:08.318496 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:08.318499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:08.318617 (+ 118us) server_negotiation.cc:407] Connection header received
0926 13:14:08.318704 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:08.318705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:08.318715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:08.318791 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:08.319416 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.319900 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.320674 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:08.320870 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:08.320956 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:08.323668 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:08.323669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:08.323670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:08.323677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:08.325088 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.325463 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.325464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.325465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.325521 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:08.325728 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:08.325729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:08.325729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:08.325831 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:08.325930 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:08.326058 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 13:14:08.326131 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 13:14:08.326741 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0993654ad76b41b094b6d3b555151adc: Uninitialized: clean time has not yet been initialized
I20260926 13:14:08.327886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1024
I20260926 13:14:08.332614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1158
I20260926 13:14:08.337258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1292
I20260926 13:14:08.341609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 930
2026-09-26T13:14:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2144 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[1462](info): set up 2 sockets
Sep 26 13:14:08 dist-test-slave-f864 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260926 13:14:09.377655 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:09.501863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:09.584445 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:09.584676 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:09.584700 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:09.585989 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:09.586087 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:14:09.586115 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:09.586139 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:09.586192 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:09.586211 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:09.587888 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:09.588181 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:09.588447 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:09.593752 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:09.593776 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:09.593775 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:09.594235 1475 server_base.cc:1034] running on GCE node
I20260926 13:14:09.594493 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:09.594798 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:09.595971 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790428449595957 us; error 30 us; skew 500 ppm
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:09.598661 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:09.599154 1475 webserver.cc:460] Webserver started at http://127.0.98.62:39369/ using document root <none> and password file <none>
I20260926 13:14:09.599396 1475 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:09.599443 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:09.599581 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:09.600713 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "b5f0b370841d4f7897d846714a408668"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "503848bb6c1c5b0bedefe5e0c4b416a9"
server_key_iv: "bf98264641714d46e58429c8598a2aee"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.601101 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5f0b370841d4f7897d846714a408668"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "503848bb6c1c5b0bedefe5e0c4b416a9"
server_key_iv: "bf98264641714d46e58429c8598a2aee"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.602491 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:09.603221 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:09.603394 1475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:09.603499 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "b5f0b370841d4f7897d846714a408668"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "503848bb6c1c5b0bedefe5e0c4b416a9"
server_key_iv: "bf98264641714d46e58429c8598a2aee"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.603585 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:09.617885 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:09.619917 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:09.620110 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:09.624910 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37783
I20260926 13:14:09.624961 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37783 every 8 connection(s)
I20260926 13:14:09.625339 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:09.626014 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:09.626750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1475
I20260926 13:14:09.626861 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:09.627024 392 external_mini_cluster.cc:1428] Setting key 7a12629146367121c7c5cfcaee9e3c83
I20260926 13:14:09.629180 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: Bootstrap starting.
I20260926 13:14:09.629782 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:09.630072 1553 log.cc:824] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:09.630846 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: No bootstrap required, opened a new log
I20260926 13:14:09.633487 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0b370841d4f7897d846714a408668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37783 } }
I20260926 13:14:09.633634 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:09.633659 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f0b370841d4f7897d846714a408668, State: Initialized, Role: FOLLOWER
I20260926 13:14:09.633813 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0b370841d4f7897d846714a408668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37783 } }
I20260926 13:14:09.633896 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:09.633919 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:09.633983 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:09.634707 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0b370841d4f7897d846714a408668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37783 } }
I20260926 13:14:09.634814 1553 leader_election.cc:302] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f0b370841d4f7897d846714a408668; no voters:
I20260926 13:14:09.634986 1553 leader_election.cc:288] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:09.635059 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:09.635207 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [term 1 LEADER]: Becoming Leader. State: Replica: b5f0b370841d4f7897d846714a408668, State: Running, Role: LEADER
I20260926 13:14:09.635282 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:09.635308 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0b370841d4f7897d846714a408668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37783 } }
I20260926 13:14:09.635677 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f0b370841d4f7897d846714a408668. Latest consensus state: current_term: 1 leader_uuid: "b5f0b370841d4f7897d846714a408668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0b370841d4f7897d846714a408668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37783 } } }
I20260926 13:14:09.635778 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:09.635738 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f0b370841d4f7897d846714a408668" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f0b370841d4f7897d846714a408668" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37783 } } }
I20260926 13:14:09.635821 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:09.636101 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:09.636654 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:09.638350 1565 catalog_manager.cc:1347] Generated new cluster ID: f14f4a39db91468fa03703a7affac0a8
I20260926 13:14:09.638404 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:09.639047 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.627597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43434 (local address 127.0.98.62:37783)
0926 13:14:09.627750 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:09.627758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:09.627784 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:14:09.628210 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:09.628218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:09.628265 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:09.628381 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:09.629179 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.630143 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.631194 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.631591 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.631672 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.634755 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:09.634760 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:09.634760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:09.634774 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:09.637060 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.637477 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.637481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.637481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.637582 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.637804 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.637805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.637805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.638129 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:09.638294 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:09.638732 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 13:14:09.638856 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260926 13:14:09.644781 1565 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:09.645321 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:09.654557 1565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: Generated new TSK 0
I20260926 13:14:09.654798 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:09.689409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37783
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:09.771694 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:09.771910 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:09.771939 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:09.773129 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:09.773175 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:09.773195 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:14:09.773213 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:09.774902 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:09.775146 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:09.775400 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:09.780649 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:09.780774 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:09.780651 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:09.780824 1579 server_base.cc:1034] running on GCE node
I20260926 13:14:09.781028 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:09.781275 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:09.782423 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790428449782411 us; error 29 us; skew 500 ppm
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:09.784945 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:09.785398 1579 webserver.cc:460] Webserver started at http://127.0.98.1:38131/ using document root <none> and password file <none>
I20260926 13:14:09.785624 1579 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:09.785694 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:09.785845 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:09.786983 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "62cf8530cb1b4967bdef5f56841af1d5"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "c7c1922b5fb591129b17af20d090c52b"
server_key_iv: "c50a69c6ee236f052dba641b3dce8042"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.787362 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "62cf8530cb1b4967bdef5f56841af1d5"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "c7c1922b5fb591129b17af20d090c52b"
server_key_iv: "c50a69c6ee236f052dba641b3dce8042"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.788686 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:09.789395 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:09.789544 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:09.789642 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "62cf8530cb1b4967bdef5f56841af1d5"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "c7c1922b5fb591129b17af20d090c52b"
server_key_iv: "c50a69c6ee236f052dba641b3dce8042"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.789740 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:09.801749 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:09.803714 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:09.803886 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:09.804529 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:09.804582 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:09.804616 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:09.804662 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:09.811321 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35561
I20260926 13:14:09.811455 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35561 every 8 connection(s)
I20260926 13:14:09.811731 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:09.814430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1579
I20260926 13:14:09.814529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:09.814695 392 external_mini_cluster.cc:1428] Setting key edebb801759fbb38b13d850afabaef01
I20260926 13:14:09.815361 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.804979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40283 (local address 127.0.98.62:37783)
0926 13:14:09.805050 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:09.805052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:09.805269 (+ 217us) server_negotiation.cc:407] Connection header received
0926 13:14:09.805756 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:09.805757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:09.805770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:09.805861 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:09.806771 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.807643 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.809196 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.809564 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.809651 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.812317 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:09.812318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:09.812319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:09.812325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:09.813881 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.814282 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.814283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.814284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.814350 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.814634 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.814636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.814636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.814744 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:09.814820 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:09.815170 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 13:14:09.815245 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:14:09.815537 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.804967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.98.1:40283)
0926 13:14:09.805113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:09.805155 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:09.805660 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:09.805913 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:09.805916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:09.806372 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:09.806660 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.806668 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.807752 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.807753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.809104 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.809107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.809655 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.809656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.809761 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.810322 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:09.810332 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:09.812235 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:09.813951 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.813956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.813958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.814215 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.814420 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.814421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.814421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.814543 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.814846 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:09.814852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:09.815041 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 13:14:09.815142 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 13:14:09.815311 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260926 13:14:09.816027 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37783
I20260926 13:14:09.816125 1710 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:09.816325 1710 heartbeater.cc:511] Master 127.0.98.62:37783 requested a full tablet report, sending...
I20260926 13:14:09.816840 1506 ts_manager.cc:194] Registered new tserver with Master: 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561)
I20260926 13:14:09.817646 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40283
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:14:09.824024 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.817324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47401 (local address 127.0.98.62:37783)
0926 13:14:09.817357 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:09.817359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:09.817467 (+ 108us) server_negotiation.cc:407] Connection header received
0926 13:14:09.817520 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:09.817520 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:09.817530 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:09.817590 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:09.817978 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.818766 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.819896 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.820079 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.820144 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.821270 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:09.821271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:09.821272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:09.821278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:09.822815 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.823221 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.823222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.823223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.823279 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.823488 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.823489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.823489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.823598 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:09.823654 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:09.823891 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 13:14:09.823959 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:14:09.824091 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.817259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.98.1:47401)
0926 13:14:09.817394 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:09.817406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:09.817462 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:09.817594 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:09.817595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:09.817803 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:09.817878 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.817880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.818883 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.818884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.819801 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.819804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.820169 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.820170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.820253 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.820686 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:09.820692 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:09.821159 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:09.822908 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.822909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.822910 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.823130 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.823350 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.823351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.823351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.823420 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.823667 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:09.823668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:09.823726 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 13:14:09.823856 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 13:14:09.823964 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:09.841542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37783
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:09.923596 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:09.923864 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:09.923909 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:09.925124 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:09.925203 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:09.925239 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:14:09.925282 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:09.926874 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:09.927192 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:09.927441 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:09.932519 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:09.932572 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:09.932519 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:09.932730 1719 server_base.cc:1034] running on GCE node
I20260926 13:14:09.932953 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:09.933218 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:09.934394 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790428449934377 us; error 27 us; skew 500 ppm
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:09.937218 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:09.937783 1719 webserver.cc:460] Webserver started at http://127.0.98.2:40701/ using document root <none> and password file <none>
I20260926 13:14:09.938037 1719 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:09.938107 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:09.938346 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:09.939435 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "78f06530c7df4eef86d7ab633e642f0f"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "d0dc07b61cb05d9858bd161972650460"
server_key_iv: "79ec6b78e5340a2b99f6fe4e6b2589bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.939810 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78f06530c7df4eef86d7ab633e642f0f"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "d0dc07b61cb05d9858bd161972650460"
server_key_iv: "79ec6b78e5340a2b99f6fe4e6b2589bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.941233 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:09.942056 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:09.942286 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:09.942385 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "78f06530c7df4eef86d7ab633e642f0f"
format_stamp: "Formatted at 2026-09-26 13:14:09 on dist-test-slave-f864"
server_key: "d0dc07b61cb05d9858bd161972650460"
server_key_iv: "79ec6b78e5340a2b99f6fe4e6b2589bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:09.942473 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:09.953923 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:09.956128 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:09.956307 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:09.956907 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:09.956967 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:09.957000 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:09.957026 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:14:09 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:09.964522 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41651
I20260926 13:14:09.964811 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41651 every 8 connection(s)
I20260926 13:14:09.964967 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:09.966665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1719
I20260926 13:14:09.966763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:09.966948 392 external_mini_cluster.cc:1428] Setting key faf62d9c369a77b272973c33584f2e4a
I20260926 13:14:09.967271 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.957285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48449 (local address 127.0.98.62:37783)
0926 13:14:09.957355 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:09.957357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:09.957562 (+ 205us) server_negotiation.cc:407] Connection header received
0926 13:14:09.958055 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:09.958057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:09.958072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:09.958156 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:09.958883 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.959458 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.960492 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.960715 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.960791 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.964214 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:09.964216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:09.964216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:09.964224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:09.965862 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.966288 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.966289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.966290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.966351 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.966592 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.966594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.966594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.966750 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:09.966822 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:09.967106 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 13:14:09.967191 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:14:09.967348 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.957275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.98.2:48449)
0926 13:14:09.957414 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:09.957454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:09.957944 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:09.958281 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:09.958285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:09.958535 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:09.958784 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.958789 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.959557 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.959558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.960392 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.960394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.960811 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.960813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.960975 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.961581 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:09.961594 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:09.964100 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:09.965937 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.965940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.965942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.966157 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.966412 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.966413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.966413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.966514 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.966810 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:09.966815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:09.966950 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 13:14:09.967031 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:14:09.967160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260926 13:14:09.967841 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37783
I20260926 13:14:09.967931 1850 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:09.968102 1850 heartbeater.cc:511] Master 127.0.98.62:37783 requested a full tablet report, sending...
I20260926 13:14:09.968881 1506 ts_manager.cc:194] Registered new tserver with Master: 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651)
I20260926 13:14:09.969421 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48449
I20260926 13:14:09.976034 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.968996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.98.2:53201)
0926 13:14:09.969115 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:09.969125 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:09.969181 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:09.970124 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:09.970124 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:09.970353 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:09.970405 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.970407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.971352 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.971353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.971857 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:09.971858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.972371 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.972372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:09.972415 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.972924 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:09.972928 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:09.973233 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:09.975021 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.975021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.975022 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.975188 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.975397 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:09.975397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:09.975398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:09.975447 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:09.975790 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:09.975790 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:09.975823 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 13:14:09.975882 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:14:09.975958 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:14:09.976109 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:09.969163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53201 (local address 127.0.98.62:37783)
0926 13:14:09.969608 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:09.969610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:09.969619 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:09.969638 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:09.969639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:09.969650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:09.969706 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:09.970485 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.971258 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.971939 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:09.972283 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:09.972368 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:09.973318 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:09.973319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:09.973320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:09.973327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:09.974937 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.975263 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.975265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.975265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.975328 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:09.975510 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:09.975511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:09.975511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:09.975721 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:09.975800 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:09.975906 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 13:14:09.975979 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:09.995162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37783
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:10.078591 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:10.078812 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:10.078836 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:10.079959 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:10.079996 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:10.080016 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:14:10.080034 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:10.081511 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:10.081709 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:10.081885 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:14:10.086895 1859 server_base.cc:1034] running on GCE node
W20260926 13:14:10.086846 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:10.086834 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:10.086839 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:10.087272 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:10.087512 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:10.088675 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790428450088661 us; error 29 us; skew 500 ppm
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:10.091107 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:10.091584 1859 webserver.cc:460] Webserver started at http://127.0.98.3:36135/ using document root <none> and password file <none>
I20260926 13:14:10.091872 1859 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:10.091939 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:10.092093 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:10.093240 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18"
format_stamp: "Formatted at 2026-09-26 13:14:10 on dist-test-slave-f864"
server_key: "0d5542f25e0c82aa33edf2e206e5cf08"
server_key_iv: "1242987204614224e13056fd8bd40d5c"
server_key_version: "encryptionkey@0"
I20260926 13:14:10.093600 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18"
format_stamp: "Formatted at 2026-09-26 13:14:10 on dist-test-slave-f864"
server_key: "0d5542f25e0c82aa33edf2e206e5cf08"
server_key_iv: "1242987204614224e13056fd8bd40d5c"
server_key_version: "encryptionkey@0"
I20260926 13:14:10.095005 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:10.095721 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.095898 1859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:10.095995 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18"
format_stamp: "Formatted at 2026-09-26 13:14:10 on dist-test-slave-f864"
server_key: "0d5542f25e0c82aa33edf2e206e5cf08"
server_key_iv: "1242987204614224e13056fd8bd40d5c"
server_key_version: "encryptionkey@0"
I20260926 13:14:10.096071 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:10.107241 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:10.109019 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:10.109184 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:10.109819 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:10.109879 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.109920 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:10.109947 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:10.117419 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35477
I20260926 13:14:10.117714 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35477 every 8 connection(s)
I20260926 13:14:10.117847 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:10.120030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 1859
I20260926 13:14:10.120082 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.110198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38899 (local address 127.0.98.62:37783)
0926 13:14:10.110244 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.110247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.110538 (+ 291us) server_negotiation.cc:407] Connection header received
0926 13:14:10.111279 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.111280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.111290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.111334 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:10.112265 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.112825 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.114544 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.114754 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.114849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.117088 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:10.117090 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:10.117090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:10.117104 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:10.118725 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.119214 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.119215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.119215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.119268 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.119454 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.119454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.119455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.119553 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:10.119603 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.119889 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.119981 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:14:10.120122 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:10.120116 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.110212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.98.3:38899)
0926 13:14:10.110383 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.110431 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.111166 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.111410 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.111431 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.111841 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:10.112161 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.112168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.112929 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.112930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.114446 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.114449 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.114869 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.114871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.114989 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.115499 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:10.115508 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:10.116987 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:10.118890 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:10.118896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:10.118898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:10.119141 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:10.119312 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:10.119312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:10.119312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:10.119404 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:10.119614 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:10.119618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:10.119737 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.119819 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.119934 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":82,"threads_started":1}
I20260926 13:14:10.120306 392 external_mini_cluster.cc:1428] Setting key 277f68d87426a88019c7d8c82ccfe522
I20260926 13:14:10.120596 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37783
I20260926 13:14:10.120692 1990 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:10.120858 1990 heartbeater.cc:511] Master 127.0.98.62:37783 requested a full tablet report, sending...
I20260926 13:14:10.121361 1506 ts_manager.cc:194] Registered new tserver with Master: 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477)
I20260926 13:14:10.121829 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38899
I20260926 13:14:10.122107 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:14:10.127861 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.121761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55863 (local address 127.0.98.62:37783)
0926 13:14:10.121799 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.121801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.121918 (+ 117us) server_negotiation.cc:407] Connection header received
0926 13:14:10.121994 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.121995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.122005 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.122061 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:10.122455 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.122983 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.123639 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.123924 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.123987 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.124939 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:10.124941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:10.124942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:10.124949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:10.126622 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.127040 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.127041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.127042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.127101 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.127327 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.127328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.127329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.127443 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:10.127521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.127687 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.127752 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:14:10.127872 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.121701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.98.3:55863)
0926 13:14:10.121855 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.121865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.121941 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.122066 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.122067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.122294 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:10.122362 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.122364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.123076 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.123077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.123558 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.123559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.124040 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.124041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.124110 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.124568 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:10.124574 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:10.124862 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:10.126741 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:10.126742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:10.126742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:10.126921 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:10.127194 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:10.127195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:10.127195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:10.127259 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:10.127520 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:10.127521 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:10.127562 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.127646 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.127752 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:10.156391 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.148299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.0.98.62:37783)
0926 13:14:10.148380 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.148382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.148397 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:14:10.148430 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.148431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.148443 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.148553 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:10.149487 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.150364 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.151144 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.151350 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.151428 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.153815 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:10.153816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:10.153817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:10.153824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:10.155477 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.155788 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.155789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.155789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.155839 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.156003 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.156003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.156003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.156103 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:10.156168 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.156273 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.156327 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:14:10.158129 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:10.159817 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:10.166675 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.162143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.0.98.3:35477)
0926 13:14:10.162601 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.162607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.162618 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:10.162660 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.162662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.162681 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.162768 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.163348 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.164454 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.165548 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.166220 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.166323 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.166405 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.166530 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.166567 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":114,"threads_started":1}
I20260926 13:14:10.166661 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.162048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.0.1:49038)
0926 13:14:10.162305 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.162352 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.162420 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.162771 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.162774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.162776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.163267 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.163270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.164551 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.164552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.165450 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.165452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.166324 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.166325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.166400 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.166414 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.166514 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.166567 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":58,"thread_start_us":156,"threads_started":1}
I20260926 13:14:10.167011 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.162068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35561 (local address 127.0.0.1:40670)
0926 13:14:10.162481 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.162490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.162531 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.163377 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.163379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.163381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.163431 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.163433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.164770 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.164771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.165844 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.165846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.166790 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.166791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.166859 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.166864 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.166894 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.166935 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260926 13:14:10.167217 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.162564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58150 (local address 127.0.98.2:41651)
0926 13:14:10.162721 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.162726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.162736 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:10.162771 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.162773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.162788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.163508 (+ 720us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.163611 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.164332 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.165930 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.166686 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.166761 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.166834 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.167048 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.167093 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260926 13:14:10.167366 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.162068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41651 (local address 127.0.0.1:58150)
0926 13:14:10.162387 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.162397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.162472 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.163388 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.163389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.163391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.163432 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.163433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.164435 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.164436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.165741 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.165743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.166864 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.166864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.166918 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.166920 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.166964 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.166994 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":30,"thread_start_us":91,"threads_started":1}
I20260926 13:14:10.167502 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.162379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40670 (local address 127.0.98.1:35561)
0926 13:14:10.162650 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.162653 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.162662 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:10.162696 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.162698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.162713 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.163580 (+ 867us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.163648 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.164687 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.165915 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.166686 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.167210 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.167303 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.167330 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.167372 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":83,"threads_started":1}
I20260926 13:14:10.167559 1924 tablet_service.cc:1467] Processing CreateTablet for tablet f5684af28fe84e7abb54b496eecc7fc6 (DEFAULT_TABLE table=test-table [id=ccb38fe533ef4ffb9594c3aeefb45b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:10.167964 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5684af28fe84e7abb54b496eecc7fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.167997 1784 tablet_service.cc:1467] Processing CreateTablet for tablet f5684af28fe84e7abb54b496eecc7fc6 (DEFAULT_TABLE table=test-table [id=ccb38fe533ef4ffb9594c3aeefb45b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:10.168107 1644 tablet_service.cc:1467] Processing CreateTablet for tablet f5684af28fe84e7abb54b496eecc7fc6 (DEFAULT_TABLE table=test-table [id=ccb38fe533ef4ffb9594c3aeefb45b8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:10.168322 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5684af28fe84e7abb54b496eecc7fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.168435 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5684af28fe84e7abb54b496eecc7fc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.171203 2013 tablet_bootstrap.cc:492] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Bootstrap starting.
I20260926 13:14:10.171705 2014 tablet_bootstrap.cc:492] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Bootstrap starting.
I20260926 13:14:10.171854 2013 tablet_bootstrap.cc:654] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.172098 2015 tablet_bootstrap.cc:492] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Bootstrap starting.
I20260926 13:14:10.172184 2013 log.cc:824] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:10.172289 2014 tablet_bootstrap.cc:654] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.172621 2014 log.cc:824] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:10.172689 2015 tablet_bootstrap.cc:654] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.173019 2015 log.cc:824] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:10.173095 2013 tablet_bootstrap.cc:492] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: No bootstrap required, opened a new log
I20260926 13:14:10.173163 2013 ts_tablet_manager.cc:1397] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:10.173339 2014 tablet_bootstrap.cc:492] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: No bootstrap required, opened a new log
I20260926 13:14:10.173415 2014 ts_tablet_manager.cc:1397] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:10.173822 2015 tablet_bootstrap.cc:492] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: No bootstrap required, opened a new log
I20260926 13:14:10.173890 2015 ts_tablet_manager.cc:1397] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:10.174976 2014 raft_consensus.cc:348] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.175171 2014 raft_consensus.cc:374] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.175212 2014 raft_consensus.cc:729] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4bbd9ecfcc45a7b74c69793fc53e18, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.175349 2014 consensus_queue.cc:260] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.175242 2013 raft_consensus.cc:348] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.175606 2013 raft_consensus.cc:374] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.175642 2014 ts_tablet_manager.cc:1428] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:14:10.175657 2013 raft_consensus.cc:729] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f06530c7df4eef86d7ab633e642f0f, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.175755 1990 heartbeater.cc:503] Master 127.0.98.62:37783 was elected leader, sending a full tablet report...
I20260926 13:14:10.175769 2013 consensus_queue.cc:260] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.175889 2015 raft_consensus.cc:348] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.176031 2013 ts_tablet_manager.cc:1428] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:14:10.176040 2015 raft_consensus.cc:374] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.176074 2015 raft_consensus.cc:729] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cf8530cb1b4967bdef5f56841af1d5, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.176141 1850 heartbeater.cc:503] Master 127.0.98.62:37783 was elected leader, sending a full tablet report...
I20260926 13:14:10.176463 2015 consensus_queue.cc:260] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.176728 2015 ts_tablet_manager.cc:1428] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:14:10.176779 1710 heartbeater.cc:503] Master 127.0.98.62:37783 was elected leader, sending a full tablet report...
W20260926 13:14:10.216221 1851 tablet.cc:2369] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:14:10.312515 1711 tablet.cc:2369] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:14:10.368563 1991 tablet.cc:2369] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:14:10.384269 2021 raft_consensus.cc:482] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:10.384402 2021 raft_consensus.cc:504] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.384770 2021 leader_election.cc:288] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477), 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651)
I20260926 13:14:10.388919 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.385057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58763 (local address 127.0.98.2:41651)
0926 13:14:10.385127 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.385129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.385300 (+ 171us) server_negotiation.cc:407] Connection header received
0926 13:14:10.385374 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.385375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.385387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.385455 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.385694 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.386523 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.388103 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.388486 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.388590 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.388602 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.388790 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.388834 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:14:10.388937 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.384980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41651 (local address 127.0.98.1:58763)
0926 13:14:10.385217 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.385228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.385309 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.385459 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.385460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.385462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.385548 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.385550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.386608 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.386609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.387976 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.387979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.388595 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.388596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.388674 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.388679 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.388778 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.388830 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":70,"thread_start_us":60,"threads_started":1}
I20260926 13:14:10.389480 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.384953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.98.1:47093)
0926 13:14:10.385110 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.385124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.385198 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.385381 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.385383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.385385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.385500 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.385502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.386641 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.386642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.387977 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.387979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.389289 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.389294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.389350 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.389354 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.389397 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.389426 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":50,"thread_start_us":92,"threads_started":1}
I20260926 13:14:10.389601 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.385047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47093 (local address 127.0.98.3:35477)
0926 13:14:10.385115 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.385118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.385197 (+ 79us) server_negotiation.cc:407] Connection header received
0926 13:14:10.385292 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.385293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.385304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.385375 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.385649 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.386529 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.388089 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.389051 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.389302 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.389315 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.389414 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.389487 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:14:10.390044 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5684af28fe84e7abb54b496eecc7fc6" candidate_uuid: "62cf8530cb1b4967bdef5f56841af1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" is_pre_election: true
I20260926 13:14:10.390272 1944 raft_consensus.cc:2454] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62cf8530cb1b4967bdef5f56841af1d5 in term 0.
I20260926 13:14:10.390616 1597 leader_election.cc:302] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4bbd9ecfcc45a7b74c69793fc53e18, 62cf8530cb1b4967bdef5f56841af1d5; no voters:
I20260926 13:14:10.390872 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5684af28fe84e7abb54b496eecc7fc6" candidate_uuid: "62cf8530cb1b4967bdef5f56841af1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f06530c7df4eef86d7ab633e642f0f" is_pre_election: true
I20260926 13:14:10.391077 1804 raft_consensus.cc:2454] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62cf8530cb1b4967bdef5f56841af1d5 in term 0.
I20260926 13:14:10.391893 2021 raft_consensus.cc:2781] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:10.391978 2021 raft_consensus.cc:482] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:10.392009 2021 raft_consensus.cc:3037] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.392850 2021 raft_consensus.cc:504] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.393077 2021 leader_election.cc:288] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [CANDIDATE]: Term 1 election: Requested vote from peers 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477), 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651)
I20260926 13:14:10.393138 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5684af28fe84e7abb54b496eecc7fc6" candidate_uuid: "62cf8530cb1b4967bdef5f56841af1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18"
I20260926 13:14:10.393235 1944 raft_consensus.cc:3037] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.394268 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5684af28fe84e7abb54b496eecc7fc6" candidate_uuid: "62cf8530cb1b4967bdef5f56841af1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f06530c7df4eef86d7ab633e642f0f"
I20260926 13:14:10.394351 1804 raft_consensus.cc:3037] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.395030 1804 raft_consensus.cc:2454] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62cf8530cb1b4967bdef5f56841af1d5 in term 1.
I20260926 13:14:10.394136 1944 raft_consensus.cc:2454] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 62cf8530cb1b4967bdef5f56841af1d5 in term 1.
I20260926 13:14:10.395218 1596 leader_election.cc:302] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62cf8530cb1b4967bdef5f56841af1d5, 78f06530c7df4eef86d7ab633e642f0f; no voters:
I20260926 13:14:10.395323 2021 raft_consensus.cc:2781] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:10.395452 2021 raft_consensus.cc:686] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 LEADER]: Becoming Leader. State: Replica: 62cf8530cb1b4967bdef5f56841af1d5, State: Running, Role: LEADER
I20260926 13:14:10.395570 2021 consensus_queue.cc:237] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } }
I20260926 13:14:10.396410 1506 catalog_manager.cc:5495] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "62cf8530cb1b4967bdef5f56841af1d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:10.455726 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.453193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.98.1:35561)
0926 13:14:10.453265 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.453266 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.453278 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:14:10.453331 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.453332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.453352 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.453408 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:10.453549 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.454019 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.454852 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.455042 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.455141 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.455363 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.455463 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.455539 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.455583 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.455603 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:14:10.457718 1944 raft_consensus.cc:1261] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Refusing update from remote peer 62cf8530cb1b4967bdef5f56841af1d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:10.457718 1804 raft_consensus.cc:1261] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Refusing update from remote peer 62cf8530cb1b4967bdef5f56841af1d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:10.458139 2024 consensus_queue.cc:1035] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:10.458484 2021 consensus_queue.cc:1035] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:10.464634 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:10.464730 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:10.465552 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: Sending DeleteTablet for 3 replicas of tablet f5684af28fe84e7abb54b496eecc7fc6
I20260926 13:14:10.465898 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet f5684af28fe84e7abb54b496eecc7fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49038
I20260926 13:14:10.465922 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet f5684af28fe84e7abb54b496eecc7fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40670
I20260926 13:14:10.465937 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet f5684af28fe84e7abb54b496eecc7fc6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58150
I20260926 13:14:10.466327 2038 tablet_replica.cc:330] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: stopping tablet replica
I20260926 13:14:10.466437 2038 raft_consensus.cc:2229] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:10.466553 2038 raft_consensus.cc:2258] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:10.466586 2039 tablet_replica.cc:330] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: stopping tablet replica
I20260926 13:14:10.466538 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:10.466683 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:10.466719 2037 tablet_replica.cc:330] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: stopping tablet replica
I20260926 13:14:10.466835 2037 raft_consensus.cc:2229] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:10.467240 2037 raft_consensus.cc:2258] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:10.467497 2038 ts_tablet_manager.cc:1905] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:10.467887 2037 ts_tablet_manager.cc:1905] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:10.468421 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 52de6c04ff9249dfbcdd3d88c97d678a (DEFAULT_TABLE table=test-table [id=6f6c5888188740ab86b0572584ecd616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:10.468529 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52de6c04ff9249dfbcdd3d88c97d678a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.469333 2038 ts_tablet_manager.cc:1918] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:10.469399 2038 log.cc:1196] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/f5684af28fe84e7abb54b496eecc7fc6
I20260926 13:14:10.469564 2037 ts_tablet_manager.cc:1918] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:10.469625 2037 log.cc:1196] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/f5684af28fe84e7abb54b496eecc7fc6
I20260926 13:14:10.469672 2038 ts_tablet_manager.cc:1939] T f5684af28fe84e7abb54b496eecc7fc6 P 62cf8530cb1b4967bdef5f56841af1d5: Deleting consensus metadata
I20260926 13:14:10.469762 2014 tablet_bootstrap.cc:492] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Bootstrap starting.
I20260926 13:14:10.469897 2037 ts_tablet_manager.cc:1939] T f5684af28fe84e7abb54b496eecc7fc6 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Deleting consensus metadata
I20260926 13:14:10.470089 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 52de6c04ff9249dfbcdd3d88c97d678a (DEFAULT_TABLE table=test-table [id=6f6c5888188740ab86b0572584ecd616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:10.470211 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52de6c04ff9249dfbcdd3d88c97d678a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.470323 1493 catalog_manager.cc:4841] TS 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477): tablet f5684af28fe84e7abb54b496eecc7fc6 (table test-table [id=ccb38fe533ef4ffb9594c3aeefb45b8f]) successfully deleted
I20260926 13:14:10.470491 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 52de6c04ff9249dfbcdd3d88c97d678a (DEFAULT_TABLE table=test-table [id=6f6c5888188740ab86b0572584ecd616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:10.470597 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52de6c04ff9249dfbcdd3d88c97d678a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.470638 2014 tablet_bootstrap.cc:654] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.470969 2039 raft_consensus.cc:2229] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:10.471081 2039 raft_consensus.cc:2258] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:10.471246 2014 tablet_bootstrap.cc:492] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: No bootstrap required, opened a new log
I20260926 13:14:10.471287 2014 ts_tablet_manager.cc:1397] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:14:10.471439 2014 raft_consensus.cc:348] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.471522 2014 raft_consensus.cc:374] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.471553 2014 raft_consensus.cc:729] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4bbd9ecfcc45a7b74c69793fc53e18, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.471601 2014 consensus_queue.cc:260] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.471673 2014 ts_tablet_manager.cc:1428] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.471750 2015 tablet_bootstrap.cc:492] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Bootstrap starting.
I20260926 13:14:10.472002 2013 tablet_bootstrap.cc:492] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Bootstrap starting.
I20260926 13:14:10.472136 2015 tablet_bootstrap.cc:654] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.472384 2013 tablet_bootstrap.cc:654] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.472846 2015 tablet_bootstrap.cc:492] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: No bootstrap required, opened a new log
I20260926 13:14:10.472888 2015 ts_tablet_manager.cc:1397] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:10.473033 2015 raft_consensus.cc:348] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.473125 2015 raft_consensus.cc:374] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.473119 1491 catalog_manager.cc:4841] TS 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561): tablet f5684af28fe84e7abb54b496eecc7fc6 (table test-table [id=ccb38fe533ef4ffb9594c3aeefb45b8f]) successfully deleted
I20260926 13:14:10.473150 2015 raft_consensus.cc:729] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cf8530cb1b4967bdef5f56841af1d5, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.473213 2015 consensus_queue.cc:260] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.473296 2015 ts_tablet_manager.cc:1428] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.473767 2039 ts_tablet_manager.cc:1905] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:10.475173 2039 ts_tablet_manager.cc:1918] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:10.475240 2013 tablet_bootstrap.cc:492] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: No bootstrap required, opened a new log
I20260926 13:14:10.475301 2013 ts_tablet_manager.cc:1397] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 13:14:10.475250 2039 log.cc:1196] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/f5684af28fe84e7abb54b496eecc7fc6
I20260926 13:14:10.475471 2013 raft_consensus.cc:348] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.475553 2013 raft_consensus.cc:374] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.475579 2013 raft_consensus.cc:729] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f06530c7df4eef86d7ab633e642f0f, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.475595 2039 ts_tablet_manager.cc:1939] T f5684af28fe84e7abb54b496eecc7fc6 P 78f06530c7df4eef86d7ab633e642f0f: Deleting consensus metadata
I20260926 13:14:10.475631 2013 consensus_queue.cc:260] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.475742 2013 ts_tablet_manager.cc:1428] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:10.475993 1492 catalog_manager.cc:4841] TS 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651): tablet f5684af28fe84e7abb54b496eecc7fc6 (table test-table [id=ccb38fe533ef4ffb9594c3aeefb45b8f]) successfully deleted
I20260926 13:14:10.490231 2020 raft_consensus.cc:482] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:10.490345 2020 raft_consensus.cc:504] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.490705 2020 leader_election.cc:288] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561), 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477)
I20260926 13:14:10.494364 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.490918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.98.2:51747)
0926 13:14:10.491083 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.491096 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.491153 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.491349 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.491350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.491352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.491420 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.491422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.492189 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.492190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.493423 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.493426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.493997 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.493998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.494105 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.494109 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.494229 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.494272 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260926 13:14:10.494374 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.491026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51747 (local address 127.0.98.3:35477)
0926 13:14:10.491075 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.491077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.491182 (+ 105us) server_negotiation.cc:407] Connection header received
0926 13:14:10.491233 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.491234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.491245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.491295 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.491487 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.492011 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.493537 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.493904 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.493981 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.493993 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.494241 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.494289 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:14:10.494623 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52de6c04ff9249dfbcdd3d88c97d678a" candidate_uuid: "78f06530c7df4eef86d7ab633e642f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" is_pre_election: true
I20260926 13:14:10.494719 1944 raft_consensus.cc:2454] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f06530c7df4eef86d7ab633e642f0f in term 0.
I20260926 13:14:10.495060 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.491119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47101 (local address 127.0.98.1:35561)
0926 13:14:10.491166 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.491168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.491294 (+ 126us) server_negotiation.cc:407] Connection header received
0926 13:14:10.491397 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.491398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.491407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.491436 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.491639 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.492383 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.494083 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.494708 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.494776 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.494786 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.494965 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.494999 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:14:10.495076 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.490918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35561 (local address 127.0.98.2:47101)
0926 13:14:10.491239 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.491248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.491317 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.491499 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.491500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.491502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.491564 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.491566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.492587 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.492587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.494000 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.494003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.494819 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.494820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.494888 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.494891 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.494959 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.494995 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260926 13:14:10.495088 1737 leader_election.cc:302] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4bbd9ecfcc45a7b74c69793fc53e18, 78f06530c7df4eef86d7ab633e642f0f; no voters:
I20260926 13:14:10.495242 2020 raft_consensus.cc:2781] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:10.495272 2020 raft_consensus.cc:482] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:10.495292 2020 raft_consensus.cc:3037] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.495302 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52de6c04ff9249dfbcdd3d88c97d678a" candidate_uuid: "78f06530c7df4eef86d7ab633e642f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62cf8530cb1b4967bdef5f56841af1d5" is_pre_election: true
I20260926 13:14:10.495409 1664 raft_consensus.cc:2454] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78f06530c7df4eef86d7ab633e642f0f in term 0.
I20260926 13:14:10.496168 2020 raft_consensus.cc:504] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.496300 2020 leader_election.cc:288] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [CANDIDATE]: Term 1 election: Requested vote from peers 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561), 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477)
I20260926 13:14:10.496469 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52de6c04ff9249dfbcdd3d88c97d678a" candidate_uuid: "78f06530c7df4eef86d7ab633e642f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62cf8530cb1b4967bdef5f56841af1d5"
I20260926 13:14:10.496548 1664 raft_consensus.cc:3037] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.496817 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52de6c04ff9249dfbcdd3d88c97d678a" candidate_uuid: "78f06530c7df4eef86d7ab633e642f0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18"
I20260926 13:14:10.496896 1944 raft_consensus.cc:3037] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.497407 1664 raft_consensus.cc:2454] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f06530c7df4eef86d7ab633e642f0f in term 1.
I20260926 13:14:10.497563 1735 leader_election.cc:302] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62cf8530cb1b4967bdef5f56841af1d5, 78f06530c7df4eef86d7ab633e642f0f; no voters:
I20260926 13:14:10.497653 2020 raft_consensus.cc:2781] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:10.497749 1944 raft_consensus.cc:2454] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78f06530c7df4eef86d7ab633e642f0f in term 1.
I20260926 13:14:10.497761 2020 raft_consensus.cc:686] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 1 LEADER]: Becoming Leader. State: Replica: 78f06530c7df4eef86d7ab633e642f0f, State: Running, Role: LEADER
I20260926 13:14:10.497830 2020 consensus_queue.cc:237] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.498590 1507 catalog_manager.cc:5495] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "78f06530c7df4eef86d7ab633e642f0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } health_report { overall_health: UNKNOWN } } }
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:10.531193 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.522965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37783 (local address 127.0.0.1:43456)
0926 13:14:10.523155 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.523168 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.523247 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.523396 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.523397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.523818 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:10.523876 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.523879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.524553 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.524554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.525356 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.525359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.525700 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.525701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.525772 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.526461 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:10.526466 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:10.528129 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:10.529842 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:10.529851 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:10.529853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:10.530114 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:10.530423 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:10.530424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:10.530424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:10.530496 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:10.530776 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:10.530783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:10.530837 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.530911 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.531059 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260926 13:14:10.531291 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.523028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.0.98.62:37783)
0926 13:14:10.523110 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.523113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.523243 (+ 130us) server_negotiation.cc:407] Connection header received
0926 13:14:10.523307 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.523308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.523318 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.523370 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:10.523968 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.524469 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.525442 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.525620 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.525690 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.528224 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:10.528226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:10.528226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:10.528233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:10.529752 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.530279 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.530281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.530281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.530350 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.530564 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.530572 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.530573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.530696 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:10.530765 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.530939 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.531114 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":45}
I20260926 13:14:10.533717 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43456:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:14:10.533964 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:10.535570 1644 tablet_service.cc:1467] Processing CreateTablet for tablet a1015bef121646269be5ea43f1510fe1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35e3c5ad407e413bb829e5a1fef64a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:14:10.535784 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1015bef121646269be5ea43f1510fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.536966 2015 tablet_bootstrap.cc:492] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: Bootstrap starting.
I20260926 13:14:10.537086 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a1015bef121646269be5ea43f1510fe1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35e3c5ad407e413bb829e5a1fef64a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:14:10.537245 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1015bef121646269be5ea43f1510fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.537408 2015 tablet_bootstrap.cc:654] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.538149 2015 tablet_bootstrap.cc:492] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: No bootstrap required, opened a new log
I20260926 13:14:10.538246 2015 ts_tablet_manager.cc:1397] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:10.538412 2015 raft_consensus.cc:348] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.538497 2015 raft_consensus.cc:374] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.538525 2015 raft_consensus.cc:729] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62cf8530cb1b4967bdef5f56841af1d5, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.538558 2015 consensus_queue.cc:260] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.538661 2015 ts_tablet_manager.cc:1428] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.538693 2032 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.538806 2032 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.539281 1784 tablet_service.cc:1467] Processing CreateTablet for tablet a1015bef121646269be5ea43f1510fe1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=35e3c5ad407e413bb829e5a1fef64a5f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:14:10.539436 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1015bef121646269be5ea43f1510fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:10.540805 2013 tablet_bootstrap.cc:492] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: Bootstrap starting.
I20260926 13:14:10.540949 2014 tablet_bootstrap.cc:492] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Bootstrap starting.
I20260926 13:14:10.541296 2013 tablet_bootstrap.cc:654] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.541401 2014 tablet_bootstrap.cc:654] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:10.542096 2013 tablet_bootstrap.cc:492] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: No bootstrap required, opened a new log
I20260926 13:14:10.542196 2013 ts_tablet_manager.cc:1397] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:10.542380 2013 raft_consensus.cc:348] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.542487 2013 raft_consensus.cc:374] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.542506 2013 raft_consensus.cc:729] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78f06530c7df4eef86d7ab633e642f0f, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.542574 2013 consensus_queue.cc:260] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.542688 2013 ts_tablet_manager.cc:1428] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.542735 2020 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.542824 2020 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.543219 2014 tablet_bootstrap.cc:492] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: No bootstrap required, opened a new log
I20260926 13:14:10.543286 2014 ts_tablet_manager.cc:1397] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:14:10.543471 2014 raft_consensus.cc:348] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.543550 2014 raft_consensus.cc:374] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:10.543565 2014 raft_consensus.cc:729] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a4bbd9ecfcc45a7b74c69793fc53e18, State: Initialized, Role: FOLLOWER
I20260926 13:14:10.543628 2014 consensus_queue.cc:260] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.543743 2014 ts_tablet_manager.cc:1428] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:10.543833 2019 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.543948 2019 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.726280 2019 raft_consensus.cc:482] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:10.726372 2019 raft_consensus.cc:504] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.726761 2019 leader_election.cc:288] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561), 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651)
I20260926 13:14:10.729605 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.727071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35455 (local address 127.0.98.2:41651)
0926 13:14:10.727145 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.727147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.727157 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:10.727240 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.727241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.727250 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.727325 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.727482 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.727964 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.728891 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.729251 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.729335 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.729346 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.729488 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.729523 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:14:10.729619 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.726968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52637 (local address 127.0.98.1:35561)
0926 13:14:10.727020 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.727023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.727118 (+ 95us) server_negotiation.cc:407] Connection header received
0926 13:14:10.727197 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.727198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.727206 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.727247 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:10.727423 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.727907 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.728881 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.729247 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.729350 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.729360 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.729511 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.729547 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:10.729633 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.726946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41651 (local address 127.0.98.3:35455)
0926 13:14:10.727073 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.727082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.727157 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.727328 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.727329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.727331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.727409 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.727410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.728051 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.728052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.728797 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.728799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.729334 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.729335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.729413 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.729417 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.729483 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.729525 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
I20260926 13:14:10.729802 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.726912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35561 (local address 127.0.98.3:52637)
0926 13:14:10.727053 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.727065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.727127 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.727260 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.727261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.727263 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:10.727347 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.727349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.727998 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.727999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.728783 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.728784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.729353 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.729354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.729427 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.729430 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.729493 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.729539 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":4,"thread_start_us":63,"threads_started":1}
I20260926 13:14:10.730039 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1015bef121646269be5ea43f1510fe1" candidate_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f06530c7df4eef86d7ab633e642f0f" is_pre_election: true
I20260926 13:14:10.730062 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1015bef121646269be5ea43f1510fe1" candidate_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62cf8530cb1b4967bdef5f56841af1d5" is_pre_election: true
I20260926 13:14:10.730139 1804 raft_consensus.cc:2454] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4bbd9ecfcc45a7b74c69793fc53e18 in term 0.
I20260926 13:14:10.730139 1664 raft_consensus.cc:2454] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a4bbd9ecfcc45a7b74c69793fc53e18 in term 0.
I20260926 13:14:10.730316 1875 leader_election.cc:302] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4bbd9ecfcc45a7b74c69793fc53e18, 62cf8530cb1b4967bdef5f56841af1d5; no voters:
I20260926 13:14:10.730463 2019 raft_consensus.cc:2781] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:10.730499 2019 raft_consensus.cc:482] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:10.730525 2019 raft_consensus.cc:3037] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.731184 2019 raft_consensus.cc:504] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.731338 2019 leader_election.cc:288] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [CANDIDATE]: Term 1 election: Requested vote from peers 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561), 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651)
I20260926 13:14:10.731482 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1015bef121646269be5ea43f1510fe1" candidate_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62cf8530cb1b4967bdef5f56841af1d5"
I20260926 13:14:10.731561 1664 raft_consensus.cc:3037] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.732102 1664 raft_consensus.cc:2454] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4bbd9ecfcc45a7b74c69793fc53e18 in term 1.
I20260926 13:14:10.732224 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1015bef121646269be5ea43f1510fe1" candidate_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78f06530c7df4eef86d7ab633e642f0f"
I20260926 13:14:10.732268 1875 leader_election.cc:302] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a4bbd9ecfcc45a7b74c69793fc53e18, 62cf8530cb1b4967bdef5f56841af1d5; no voters:
I20260926 13:14:10.732309 1804 raft_consensus.cc:3037] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:10.732424 2019 raft_consensus.cc:2781] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:10.732553 2019 raft_consensus.cc:686] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 LEADER]: Becoming Leader. State: Replica: 5a4bbd9ecfcc45a7b74c69793fc53e18, State: Running, Role: LEADER
I20260926 13:14:10.732625 2019 consensus_queue.cc:237] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } }
I20260926 13:14:10.732963 2058 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: TxnStatusTablet state changed. Reason: New leader 5a4bbd9ecfcc45a7b74c69793fc53e18. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.733101 1804 raft_consensus.cc:2454] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a4bbd9ecfcc45a7b74c69793fc53e18 in term 1.
I20260926 13:14:10.733122 2058 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:14:10.733304 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:14:10.733433 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:14:10.733747 1507 catalog_manager.cc:5495] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:10.774328 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.771394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.0.1:49054)
0926 13:14:10.771480 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.771493 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.771540 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.771730 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.771731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.771732 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:10.771781 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.771782 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.772696 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.772696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.773276 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.773277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.773745 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.773745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.773803 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.773818 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.774124 (+ 306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.774131 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.774184 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.774258 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:14:10.774317 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.771521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49054 (local address 127.0.98.3:35477)
0926 13:14:10.771566 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.771568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.771577 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:10.771616 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.771617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.771639 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.771737 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:10.771850 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.772611 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.773359 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.773658 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.773741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.773912 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.774052 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.774135 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.774190 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.774212 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:10.775655 1804 raft_consensus.cc:1261] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Refusing update from remote peer 5a4bbd9ecfcc45a7b74c69793fc53e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:10.775807 1664 raft_consensus.cc:1261] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Refusing update from remote peer 5a4bbd9ecfcc45a7b74c69793fc53e18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:10.776088 2058 consensus_queue.cc:1035] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:10.776235 2019 consensus_queue.cc:1035] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:10.776767 2020 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: TxnStatusTablet state changed. Reason: New leader 5a4bbd9ecfcc45a7b74c69793fc53e18. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.776877 2020 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.776981 2032 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: TxnStatusTablet state changed. Reason: New leader 5a4bbd9ecfcc45a7b74c69793fc53e18. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.777073 2032 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.777765 2058 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.777848 2058 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:14:10.778147 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333594934374944768 to 7333594934201421824. Current Snapshot: MvccSnapshot[applied={T|T < 7333594934374944768}]
I20260926 13:14:10.778283 2020 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.778369 2020 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 78f06530c7df4eef86d7ab633e642f0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.779933 2032 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.780028 2032 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 62cf8530cb1b4967bdef5f56841af1d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:10.780690 2058 tablet_replica.cc:439] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 } } peers { permanent_uuid: "78f06530c7df4eef86d7ab633e642f0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41651 } } peers { permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 } } }
I20260926 13:14:10.780795 2058 tablet_replica.cc:442] T a1015bef121646269be5ea43f1510fe1 P 5a4bbd9ecfcc45a7b74c69793fc53e18: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:14:10.786082 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.779358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.0.98.2:41651)
0926 13:14:10.779434 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.779436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.779443 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:10.779476 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.779477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.779495 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.779574 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:10.779832 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.780489 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.781332 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.781614 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.781688 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.781957 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.782110 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.782203 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.785937 (+ 3734us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.785970 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 13:14:10.795766 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.788786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38639 (local address 127.0.98.3:35477)
0926 13:14:10.793256 (+ 4470us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.793259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.793271 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:14:10.793298 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.793299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.793310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.793403 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:10.793592 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.794087 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.794948 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.795149 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.795243 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.795461 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.795504 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.795581 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.795652 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.795677 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4425}
I20260926 13:14:10.795804 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.788748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35477 (local address 127.0.98.2:38639)
0926 13:14:10.788894 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.788906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.788961 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.793413 (+ 4452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.793414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.793416 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:10.793508 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.793511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.794203 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.794204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.794831 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.794834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.795263 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.795264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.795330 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.795349 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.795585 (+ 236us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.795588 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.795647 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.795694 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260926 13:14:10.799147 1664 raft_consensus.cc:1261] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Refusing update from remote peer 78f06530c7df4eef86d7ab633e642f0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:10.799147 1944 raft_consensus.cc:1261] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Refusing update from remote peer 78f06530c7df4eef86d7ab633e642f0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:10.799743 2045 consensus_queue.cc:1035] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a4bbd9ecfcc45a7b74c69793fc53e18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:10.799964 2020 consensus_queue.cc:1035] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "62cf8530cb1b4967bdef5f56841af1d5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:10.802271 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333594934470987776 to 7333594933239681024. Current Snapshot: MvccSnapshot[applied={T|T < 7333594934470987776}]
I20260926 13:14:10.803591 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333594934470987776 to 7333594933239681024. Current Snapshot: MvccSnapshot[applied={T|T < 7333594934470987776}]
I20260926 13:14:10.810998 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.807671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48499 (local address 127.0.98.2:41651)
0926 13:14:10.807720 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.807722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.807881 (+ 159us) server_negotiation.cc:407] Connection header received
0926 13:14:10.807964 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.807965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.807977 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.808039 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:10.808166 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.808752 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.810077 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.810374 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.810465 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.810676 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.810715 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.810794 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.810886 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.810915 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:10.811246 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.807632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41651 (local address 127.0.98.3:48499)
0926 13:14:10.807793 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:10.807805 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:10.807898 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:10.808041 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:10.808042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:10.808043 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:10.808093 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.808102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.808851 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.808852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.809955 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:10.809959 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.810488 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.810489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:10.810552 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.810566 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:10.810797 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:10.810799 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:14:10.811082 (+ 283us) client_negotiation.cc:240] Negotiation successful
0926 13:14:10.811137 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260926 13:14:10.821580 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:10.821715 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:10.822360 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5f0b370841d4f7897d846714a408668: Sending DeleteTablet for 3 replicas of tablet 52de6c04ff9249dfbcdd3d88c97d678a
I20260926 13:14:10.822649 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 52de6c04ff9249dfbcdd3d88c97d678a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58150
I20260926 13:14:10.822657 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 52de6c04ff9249dfbcdd3d88c97d678a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49038
I20260926 13:14:10.822747 2039 tablet_replica.cc:330] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: stopping tablet replica
I20260926 13:14:10.822770 2037 tablet_replica.cc:330] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: stopping tablet replica
I20260926 13:14:10.822795 2039 raft_consensus.cc:2229] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:10.822809 2037 raft_consensus.cc:2229] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:10.822842 2037 raft_consensus.cc:2258] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:10.822885 2039 raft_consensus.cc:2258] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:10.823179 2037 ts_tablet_manager.cc:1905] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:10.823165 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 52de6c04ff9249dfbcdd3d88c97d678a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40670
I20260926 13:14:10.823251 2038 tablet_replica.cc:330] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: stopping tablet replica
I20260926 13:14:10.823318 2038 raft_consensus.cc:2229] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:10.823362 2038 raft_consensus.cc:2258] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:10.823717 2038 ts_tablet_manager.cc:1905] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:10.825062 2037 ts_tablet_manager.cc:1918] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:10.825131 2037 log.cc:1196] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/52de6c04ff9249dfbcdd3d88c97d678a
I20260926 13:14:10.825348 2037 ts_tablet_manager.cc:1939] T 52de6c04ff9249dfbcdd3d88c97d678a P 5a4bbd9ecfcc45a7b74c69793fc53e18: Deleting consensus metadata
I20260926 13:14:10.825824 1493 catalog_manager.cc:4841] TS 5a4bbd9ecfcc45a7b74c69793fc53e18 (127.0.98.3:35477): tablet 52de6c04ff9249dfbcdd3d88c97d678a (table test-table [id=6f6c5888188740ab86b0572584ecd616]) successfully deleted
I20260926 13:14:10.826056 2039 ts_tablet_manager.cc:1905] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:10.828123 2039 ts_tablet_manager.cc:1918] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:10.828193 2039 log.cc:1196] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/52de6c04ff9249dfbcdd3d88c97d678a
I20260926 13:14:10.828420 2039 ts_tablet_manager.cc:1939] T 52de6c04ff9249dfbcdd3d88c97d678a P 78f06530c7df4eef86d7ab633e642f0f: Deleting consensus metadata
I20260926 13:14:10.828603 2038 ts_tablet_manager.cc:1918] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:10.828668 2038 log.cc:1196] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/52de6c04ff9249dfbcdd3d88c97d678a
I20260926 13:14:10.828886 2038 ts_tablet_manager.cc:1939] T 52de6c04ff9249dfbcdd3d88c97d678a P 62cf8530cb1b4967bdef5f56841af1d5: Deleting consensus metadata
I20260926 13:14:10.828997 1492 catalog_manager.cc:4841] TS 78f06530c7df4eef86d7ab633e642f0f (127.0.98.2:41651): tablet 52de6c04ff9249dfbcdd3d88c97d678a (table test-table [id=6f6c5888188740ab86b0572584ecd616]) successfully deleted
I20260926 13:14:10.829227 1491 catalog_manager.cc:4841] TS 62cf8530cb1b4967bdef5f56841af1d5 (127.0.98.1:35561): tablet 52de6c04ff9249dfbcdd3d88c97d678a (table test-table [id=6f6c5888188740ab86b0572584ecd616]) successfully deleted
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428450, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:10.845348 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.830451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.0.98.1:35561)
0926 13:14:10.830504 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.830506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.834409 (+ 3903us) server_negotiation.cc:407] Connection header received
0926 13:14:10.834563 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.834564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.834592 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.834623 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:10.835520 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.836330 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.837552 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.837870 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.837913 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.841730 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:10.841744 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:10.841745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:10.841754 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:10.843851 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.844277 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.844280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.844280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.844342 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.844589 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.844590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.844590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.844770 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:10.844897 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.845155 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.845246 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 13:14:10.846226 1690 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40684
I20260926 13:14:10.855281 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:10.847483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.0.98.62:37783)
0926 13:14:10.847536 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:10.847538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:10.847641 (+ 103us) server_negotiation.cc:407] Connection header received
0926 13:14:10.847738 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:10.847739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:10.847751 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:10.847781 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:10.848485 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.849287 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.850399 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:10.850700 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:10.850743 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:10.852275 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:10.852277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:10.852277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:10.852284 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:10.853994 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.854441 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.854443 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.854443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.854499 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:10.854692 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:10.854693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:10.854693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:10.854831 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:10.854903 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:10.855102 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:14:10.855181 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 13:14:10.856202 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43466
I20260926 13:14:10.857170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1579
I20260926 13:14:10.866209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1719
I20260926 13:14:10.875954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1859
I20260926 13:14:10.881204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 1475
2026-09-26T13:14:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2538 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[2083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[2083](info): set up 2 sockets
Sep 26 13:14:10 dist-test-slave-f864 krb5kdc[2083](info): commencing operation
krb5kdc: starting...
W20260926 13:14:11.913826 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:11 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:12.033670 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39625
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:12.115343 2096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:12.115574 2096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:12.115598 2096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:12.116755 2096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:12.116793 2096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:14:12.116807 2096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:12.116820 2096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:12.116832 2096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:12.116844 2096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:12.118279 2096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39625
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:12.118468 2096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:12.118654 2096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:12.123735 2101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:12.123741 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:12.124084 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:12.124079 2096 server_base.cc:1034] running on GCE node
I20260926 13:14:12.124383 2096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:12.124656 2096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:12.125825 2096 hybrid_clock.cc:648] HybridClock initialized: now 1790428452125809 us; error 55 us; skew 500 ppm
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:12.128553 2096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:12.128986 2096 webserver.cc:460] Webserver started at http://127.0.98.62:42697/ using document root <none> and password file <none>
I20260926 13:14:12.129221 2096 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:12.129288 2096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:12.129446 2096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:12.130584 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "28f406e81b124600b181ce549f01e3d3"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "2dfd0763b48319d3e1d0bc85c16cf18b"
server_key_iv: "c21fce8c54cffe2940cc6b7a132bb937"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.130949 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "28f406e81b124600b181ce549f01e3d3"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "2dfd0763b48319d3e1d0bc85c16cf18b"
server_key_iv: "c21fce8c54cffe2940cc6b7a132bb937"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.132230 2096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:12.132982 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.133144 2096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:12.133249 2096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "28f406e81b124600b181ce549f01e3d3"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "2dfd0763b48319d3e1d0bc85c16cf18b"
server_key_iv: "c21fce8c54cffe2940cc6b7a132bb937"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.133320 2096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:12.150897 2096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:12.153396 2096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:12.153570 2096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:12.158308 2096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39625
I20260926 13:14:12.158381 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39625 every 8 connection(s)
I20260926 13:14:12.158783 2096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:12.159469 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.161751 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Bootstrap starting.
I20260926 13:14:12.162400 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.162696 2174 log.cc:824] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:12.163365 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: No bootstrap required, opened a new log
I20260926 13:14:12.165426 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f406e81b124600b181ce549f01e3d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39625 } }
I20260926 13:14:12.165565 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.165587 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f406e81b124600b181ce549f01e3d3, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.165769 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f406e81b124600b181ce549f01e3d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39625 } }
I20260926 13:14:12.165834 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:12.165856 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:12.165907 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:12.166522 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f406e81b124600b181ce549f01e3d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39625 } }
I20260926 13:14:12.166635 2174 leader_election.cc:302] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28f406e81b124600b181ce549f01e3d3; no voters:
I20260926 13:14:12.166800 2174 leader_election.cc:288] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:12.166822 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:12.167033 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [term 1 LEADER]: Becoming Leader. State: Replica: 28f406e81b124600b181ce549f01e3d3, State: Running, Role: LEADER
I20260926 13:14:12.167090 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:12.167199 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f406e81b124600b181ce549f01e3d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39625 } }
I20260926 13:14:12.167600 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28f406e81b124600b181ce549f01e3d3. Latest consensus state: current_term: 1 leader_uuid: "28f406e81b124600b181ce549f01e3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f406e81b124600b181ce549f01e3d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39625 } } }
I20260926 13:14:12.167695 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:12.167595 2178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28f406e81b124600b181ce549f01e3d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28f406e81b124600b181ce549f01e3d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39625 } } }
I20260926 13:14:12.167757 2178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:12.167954 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:12.168493 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:12.168637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 2096
I20260926 13:14:12.168730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:12.168905 392 external_mini_cluster.cc:1428] Setting key 07d72d499ea933f9cbfa96afeb46dba1
I20260926 13:14:12.170198 2186 catalog_manager.cc:1347] Generated new cluster ID: 4b83e09641534edea81beef6fe8d9abf
I20260926 13:14:12.170245 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:12.180447 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:12.180461 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.169341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.98.62:39625)
0926 13:14:12.169498 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.169507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.169692 (+ 185us) server_negotiation.cc:407] Connection header received
0926 13:14:12.169836 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.169842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.169886 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.170033 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.170748 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.172137 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.173173 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.173808 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.173921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.176669 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.176672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.176673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.176685 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.178689 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.179092 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.179095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.179095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.179212 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.179432 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.179433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.179433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.179719 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.179895 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.180003 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.180171 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260926 13:14:12.181178 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:12.187598 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Generated new TSK 0
I20260926 13:14:12.187778 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:12.221148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39625
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:12.316766 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:12.316984 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:12.317011 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:12.318228 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:12.318269 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:12.318290 2200 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:14:12.318303 2200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:14:12.318367 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:12.319953 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:12.320226 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:12.320464 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:12.325615 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:12.325618 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:12.325644 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:12.325845 2200 server_base.cc:1034] running on GCE node
I20260926 13:14:12.326095 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:12.326385 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:12.327556 2200 hybrid_clock.cc:648] HybridClock initialized: now 1790428452327535 us; error 34 us; skew 500 ppm
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:12.330086 2200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:12.330545 2200 webserver.cc:460] Webserver started at http://127.0.98.1:41381/ using document root <none> and password file <none>
I20260926 13:14:12.330777 2200 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:12.330848 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:12.331012 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:12.332084 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "0c15312690b115ad4e492738236e0add"
server_key_iv: "9069381e73d34fc0f5cde9941de7f08a"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.332455 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "0c15312690b115ad4e492738236e0add"
server_key_iv: "9069381e73d34fc0f5cde9941de7f08a"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.333774 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:12.334564 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.334743 2200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.334841 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "0c15312690b115ad4e492738236e0add"
server_key_iv: "9069381e73d34fc0f5cde9941de7f08a"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.334934 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:12.360018 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:12.362049 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:12.362287 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:12.362834 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:12.362908 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.362957 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:12.362983 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:12.370091 2200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32907
I20260926 13:14:12.370179 2330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32907 every 8 connection(s)
I20260926 13:14:12.370462 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:14:12.373028 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.363442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57263 (local address 127.0.98.62:39625)
0926 13:14:12.363508 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.363510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.363712 (+ 202us) server_negotiation.cc:407] Connection header received
0926 13:14:12.364289 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.364290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.364308 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.364396 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.365317 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.366231 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.367474 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.367789 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.367859 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.370595 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.370596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.370596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.370605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.371894 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.372248 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.372249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.372249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.372298 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.372515 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.372516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.372516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.372615 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.372665 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.372886 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.372953 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:14:12.373208 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.363422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.98.1:57263)
0926 13:14:12.363591 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.363626 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.364147 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.364491 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.364500 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.364904 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.365210 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.365216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.366330 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.366331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.367371 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.367374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.367881 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.367882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.368030 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.368681 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.368690 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.370489 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.371965 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.371970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.371972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.372184 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.372354 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.372355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.372355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.372461 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.372669 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.372673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.372778 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.372857 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.372971 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260926 13:14:12.373785 2331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39625
I20260926 13:14:12.373859 2331 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:12.374055 2331 heartbeater.cc:511] Master 127.0.98.62:39625 requested a full tablet report, sending...
I20260926 13:14:12.374549 2128 ts_manager.cc:194] Registered new tserver with Master: 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907)
I20260926 13:14:12.375262 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57263
I20260926 13:14:12.377099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 2200
I20260926 13:14:12.377197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:12.377410 392 external_mini_cluster.cc:1428] Setting key 263f1b0cba9b3f8764630d12094420f7
I20260926 13:14:12.380206 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.374853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35825 (local address 127.0.98.62:39625)
0926 13:14:12.374891 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.374893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.375004 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:14:12.375085 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.375086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.375096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.375140 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.375475 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.375986 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.376626 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.376795 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.376843 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.377715 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.377716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.377717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.377723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.379121 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.379479 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.379480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.379480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.379531 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.379734 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.379735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.379735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.379841 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.379913 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.380046 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.380121 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:14:12.380221 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.374800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.98.1:35825)
0926 13:14:12.374928 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.374938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.375028 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.375153 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.375154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.375318 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.375384 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.375386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.376086 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.376087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.376559 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.376560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.376856 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.376857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.376926 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.377341 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.377345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.377661 (+ 316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.379221 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.379222 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.379222 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.379387 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.379604 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.379605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.379605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.379661 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.379906 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.379907 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.379950 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.380019 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.380121 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:12.405040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39625
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:12.486099 2340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:12.486341 2340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:12.486367 2340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:12.487531 2340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:12.487573 2340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:12.487593 2340 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:14:12.487648 2340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:14:12.487673 2340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:12.489120 2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:12.489364 2340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:12.489595 2340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:12.494623 2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:12.494658 2340 server_base.cc:1034] running on GCE node
W20260926 13:14:12.494620 2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:12.494616 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:12.495083 2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:12.495343 2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:12.496510 2340 hybrid_clock.cc:648] HybridClock initialized: now 1790428452496506 us; error 38 us; skew 500 ppm
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:12.499004 2340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:12.499523 2340 webserver.cc:460] Webserver started at http://127.0.98.2:35091/ using document root <none> and password file <none>
I20260926 13:14:12.499745 2340 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:12.499828 2340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:12.499979 2340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:12.501006 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "57db91d613c04eb0b5bfdcd028da811a"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "1173644145b3bae33aa36de0feb74d5d"
server_key_iv: "7dbc2924072a4aaa6505a8ff7282fae9"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.501379 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "57db91d613c04eb0b5bfdcd028da811a"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "1173644145b3bae33aa36de0feb74d5d"
server_key_iv: "7dbc2924072a4aaa6505a8ff7282fae9"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.502789 2340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:14:12.503571 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.503789 2340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:12.503909 2340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "57db91d613c04eb0b5bfdcd028da811a"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "1173644145b3bae33aa36de0feb74d5d"
server_key_iv: "7dbc2924072a4aaa6505a8ff7282fae9"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.503988 2340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:12.534600 2340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:12.536760 2340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:12.536954 2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:12.537568 2340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:12.537631 2340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.537667 2340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:12.537694 2340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:12.545176 2340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44277
I20260926 13:14:12.545537 2340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:12.545660 2470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44277 every 8 connection(s)
I20260926 13:14:12.548321 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.538096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47443 (local address 127.0.98.62:39625)
0926 13:14:12.538154 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.538182 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.538477 (+ 295us) server_negotiation.cc:407] Connection header received
0926 13:14:12.539255 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.539256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.539305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.539385 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.540195 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.541040 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.542196 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.542437 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.542536 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.544907 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.544909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.544909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.544918 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.546759 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.547240 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.547241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.547242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.547320 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.547612 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.547613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.547613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.547742 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.547825 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.548122 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.548206 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:14:12.548548 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.538088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.98.2:47443)
0926 13:14:12.538306 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.538358 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.539140 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.539514 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.539518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.539845 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.540113 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.540119 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.541151 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.541152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.542071 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.542073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.542558 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.542560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.542692 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.543271 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.543280 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.544795 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.546871 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.546878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.546880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.547151 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.547403 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.547404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.547404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.547528 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.547815 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.547821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.547975 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.548099 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.548275 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":127,"threads_started":1}
I20260926 13:14:12.549026 2471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39625
I20260926 13:14:12.549130 2471 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:12.549311 2471 heartbeater.cc:511] Master 127.0.98.62:39625 requested a full tablet report, sending...
I20260926 13:14:12.549731 2127 ts_manager.cc:194] Registered new tserver with Master: 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:12.550076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 2340
I20260926 13:14:12.550156 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:12.550292 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47443
I20260926 13:14:12.550356 392 external_mini_cluster.cc:1428] Setting key 3b594e6b6f9990c9108947cad49d6777
I20260926 13:14:12.557010 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.550271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36871 (local address 127.0.98.62:39625)
0926 13:14:12.550315 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.550317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.550709 (+ 392us) server_negotiation.cc:407] Connection header received
0926 13:14:12.550793 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.550794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.550804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.550865 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.551272 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.551781 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.552825 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.553029 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.553080 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.554052 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.554053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.554054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.554062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.555719 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.556118 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.556120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.556120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.556190 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.556408 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.556409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.556409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.556566 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.556649 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.556829 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.556910 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:14:12.557025 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.550330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.98.2:36871)
0926 13:14:12.550656 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.550668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.550725 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.550871 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.550872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.551119 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.551184 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.551186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.551881 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.551883 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.552718 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.552721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.553105 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.553106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.553179 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.553603 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.553608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.553954 (+ 346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.555835 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.555836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.555837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.556024 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.556262 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.556263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.556263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.556326 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.556641 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.556642 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.556689 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.556800 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.556910 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":192,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:12.576745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39625
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:12.657537 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:12.657755 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:12.657781 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:12.659070 2480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:12.659121 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:12.659144 2480 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 13:14:12.659157 2480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:14:12.659175 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:12.660627 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:12.660854 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:12.661075 2480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:12.666249 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:12.666337 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:12.666494 2480 server_base.cc:1034] running on GCE node
W20260926 13:14:12.666255 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:12.666785 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:12.667075 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:12.668273 2480 hybrid_clock.cc:648] HybridClock initialized: now 1790428452668257 us; error 48 us; skew 500 ppm
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:12.670892 2480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:12.671370 2480 webserver.cc:460] Webserver started at http://127.0.98.3:36017/ using document root <none> and password file <none>
I20260926 13:14:12.671594 2480 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:12.671660 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:12.671802 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:12.672847 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "86bc90a1d05c4c409b29763059c5c815"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "59af6d914e6733a44239b44707b4137e"
server_key_iv: "2068a4041e3519dacaeeef107ef8a30e"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.673221 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "86bc90a1d05c4c409b29763059c5c815"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "59af6d914e6733a44239b44707b4137e"
server_key_iv: "2068a4041e3519dacaeeef107ef8a30e"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.674703 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:12.675429 2495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.675618 2480 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:12.675719 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "86bc90a1d05c4c409b29763059c5c815"
format_stamp: "Formatted at 2026-09-26 13:14:12 on dist-test-slave-f864"
server_key: "59af6d914e6733a44239b44707b4137e"
server_key_iv: "2068a4041e3519dacaeeef107ef8a30e"
server_key_version: "encryptionkey@0"
I20260926 13:14:12.675791 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:12.686700 2480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:12.689352 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:12.689548 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:12.690117 2480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:12.690219 2480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.690284 2480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:12.690322 2480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.697546 2480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41163
I20260926 13:14:12.697572 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41163 every 8 connection(s)
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:12.697991 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:12.700861 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.690643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58145 (local address 127.0.98.62:39625)
0926 13:14:12.690719 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.690721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.691002 (+ 281us) server_negotiation.cc:407] Connection header received
0926 13:14:12.691627 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.691629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.691641 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.691696 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.692519 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.693322 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.694928 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.695240 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.695282 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.698204 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.698207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.698207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.698218 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.699644 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.700008 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.700009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.700010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.700063 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.700308 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.700309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.700309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.700428 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.700494 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.700726 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.700798 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:14:12.701014 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.690573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.98.3:58145)
0926 13:14:12.690870 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.690899 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.691510 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.691772 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.691777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.692124 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.692403 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.692409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.693438 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.693439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.694811 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.694814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.695377 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.695378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.695525 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.696118 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.696127 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.698050 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.699718 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.699723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.699725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.699937 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.700125 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.700126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.700126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.700217 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.700495 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.700499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.700611 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.700690 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.700809 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":141,"threads_started":1}
I20260926 13:14:12.701573 2611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39625
I20260926 13:14:12.701665 2611 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:12.701874 2611 heartbeater.cc:511] Master 127.0.98.62:39625 requested a full tablet report, sending...
I20260926 13:14:12.702252 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 2480
I20260926 13:14:12.702344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:12.702342 2128 ts_manager.cc:194] Registered new tserver with Master: 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163)
I20260926 13:14:12.702528 392 external_mini_cluster.cc:1428] Setting key 738547bb644d198e68139e6d2d9e3954
I20260926 13:14:12.702845 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58145
I20260926 13:14:12.703306 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:14:12.709080 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.702699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38731 (local address 127.0.98.62:39625)
0926 13:14:12.702740 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.702742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.702959 (+ 217us) server_negotiation.cc:407] Connection header received
0926 13:14:12.703051 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.703052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.703062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.703091 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.703489 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.704182 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.704855 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.705162 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.705234 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.706395 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.706397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.706397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.706404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.708014 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.708326 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.708327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.708328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.708389 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.708634 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.708634 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.708635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.708733 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.708793 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.708964 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.709018 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:14:12.709230 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.702773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.98.3:38731)
0926 13:14:12.702900 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.702911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.702983 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.703160 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.703161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.703321 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.703408 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.703410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.704283 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.704284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.704781 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.704783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.705243 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.705244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.705326 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.705853 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.705859 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.706315 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.708093 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.708094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.708094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.708252 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.708464 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.708465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.708466 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.708560 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.708804 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.708805 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.708854 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.708949 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.709066 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":40,"thread_start_us":84,"threads_started":1}
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:12.727258 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.720708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.0.98.62:39625)
0926 13:14:12.720771 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.720773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.720791 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:14:12.720819 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.720820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.720828 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.720886 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.721403 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.721872 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.722713 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.722889 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.722949 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.724997 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.724998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.724998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.725005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.726367 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.726677 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.726677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.726678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.726725 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.726881 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.726881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.726882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.726985 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.727056 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.727138 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.727195 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:14:12.728798 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:12.730516 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:12.736553 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.732722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44277 (local address 127.0.0.1:33560)
0926 13:14:12.732964 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.732974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.733028 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.733299 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.733301 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.733303 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:12.733349 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.733350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.734455 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.734456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.735277 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.735280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.736214 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.736215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.736305 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.736319 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.736417 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.736479 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":95,"threads_started":1}
I20260926 13:14:12.736568 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.732977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.0.98.2:44277)
0926 13:14:12.733155 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.733159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.733168 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:12.733198 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.733200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.733212 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.733278 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:12.733428 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.734370 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.735411 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.736126 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.736215 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.736274 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.736433 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.736469 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260926 13:14:12.737360 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 1916fcfba279420f950d690417d25cda (DEFAULT_TABLE table=test-table [id=63bc9cf16b4a4e6f8f5e023c65678bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:12.737766 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916fcfba279420f950d690417d25cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.737775 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.732944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.0.98.3:41163)
0926 13:14:12.733670 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.733673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.733682 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:12.733717 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.733719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.733736 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.733806 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:12.734002 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.735099 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.736465 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.737348 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.737430 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.737478 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.737641 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.737678 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":91,"threads_started":1}
I20260926 13:14:12.737871 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.732817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.0.98.1:32907)
0926 13:14:12.733370 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.733373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.733380 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:12.733406 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.733408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.733421 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.733531 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:12.734358 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.735409 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.736651 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.737348 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.737396 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.737453 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.737724 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.737763 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":73,"threads_started":1}
I20260926 13:14:12.737689 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.732733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41163 (local address 127.0.0.1:60946)
0926 13:14:12.733128 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.733136 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.733170 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.733872 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.733873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.733875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:12.733930 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.733931 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.735203 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.735204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.736366 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.736368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.737448 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.737450 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.737513 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.737517 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.737582 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.737621 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":82,"threads_started":1}
I20260926 13:14:12.738121 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.732719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32907 (local address 127.0.0.1:47136)
0926 13:14:12.732884 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.732944 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.733028 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.734229 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.734230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.734232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:12.734267 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.734268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.735496 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.735497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.736560 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.736561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.737515 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.737516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.737581 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.737585 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.737631 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.737672 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":55,"threads_started":1}
I20260926 13:14:12.738658 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 1916fcfba279420f950d690417d25cda (DEFAULT_TABLE table=test-table [id=63bc9cf16b4a4e6f8f5e023c65678bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:12.738727 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 1916fcfba279420f950d690417d25cda (DEFAULT_TABLE table=test-table [id=63bc9cf16b4a4e6f8f5e023c65678bc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:12.738983 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916fcfba279420f950d690417d25cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.739078 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1916fcfba279420f950d690417d25cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.741659 2634 tablet_bootstrap.cc:492] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Bootstrap starting.
I20260926 13:14:12.742149 2635 tablet_bootstrap.cc:492] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Bootstrap starting.
I20260926 13:14:12.742156 2636 tablet_bootstrap.cc:492] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Bootstrap starting.
I20260926 13:14:12.742527 2634 tablet_bootstrap.cc:654] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.742801 2635 tablet_bootstrap.cc:654] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.742937 2634 log.cc:824] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:12.743141 2636 tablet_bootstrap.cc:654] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.743188 2635 log.cc:824] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:12.743436 2636 log.cc:824] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:12.743829 2634 tablet_bootstrap.cc:492] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: No bootstrap required, opened a new log
I20260926 13:14:12.743901 2634 ts_tablet_manager.cc:1397] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:12.744134 2636 tablet_bootstrap.cc:492] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: No bootstrap required, opened a new log
I20260926 13:14:12.744201 2636 ts_tablet_manager.cc:1397] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:14:12.745884 2634 raft_consensus.cc:348] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.746060 2634 raft_consensus.cc:374] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.746109 2634 raft_consensus.cc:729] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8e73d841da451aa05ce7f7da5c67f8, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.746083 2636 raft_consensus.cc:348] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.746255 2636 raft_consensus.cc:374] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.746289 2636 raft_consensus.cc:729] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.746248 2634 consensus_queue.cc:260] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.746387 2636 consensus_queue.cc:260] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.746538 2634 ts_tablet_manager.cc:1428] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:14:12.746582 2331 heartbeater.cc:503] Master 127.0.98.62:39625 was elected leader, sending a full tablet report...
I20260926 13:14:12.746623 2636 ts_tablet_manager.cc:1428] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 13:14:12.746732 2611 heartbeater.cc:503] Master 127.0.98.62:39625 was elected leader, sending a full tablet report...
I20260926 13:14:12.747136 2635 tablet_bootstrap.cc:492] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: No bootstrap required, opened a new log
I20260926 13:14:12.747208 2635 ts_tablet_manager.cc:1397] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 13:14:12.748759 2635 raft_consensus.cc:348] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.748880 2635 raft_consensus.cc:374] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.748903 2635 raft_consensus.cc:729] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.749020 2635 consensus_queue.cc:260] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.749257 2635 ts_tablet_manager.cc:1428] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:12.749328 2471 heartbeater.cc:503] Master 127.0.98.62:39625 was elected leader, sending a full tablet report...
W20260926 13:14:12.796205 2472 tablet.cc:2369] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:14:12.845701 2642 raft_consensus.cc:482] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:12.845835 2642 raft_consensus.cc:504] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.846295 2642 leader_election.cc:288] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163)
I20260926 13:14:12.850438 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.846664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48261 (local address 127.0.98.1:32907)
0926 13:14:12.846715 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.846718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.846730 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:14:12.846820 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.846822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.846834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.846918 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:12.847121 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.847907 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.849490 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.850052 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.850118 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.850127 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.850333 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.850373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:14:12.850435 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.846602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32907 (local address 127.0.98.2:48261)
0926 13:14:12.846667 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.846682 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.846736 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.846948 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.846949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.846951 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:12.847034 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.847037 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.848051 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.848053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.849378 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.849381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.850125 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.850126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.850232 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.850237 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.850311 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.850355 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":77}
I20260926 13:14:12.850505 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.846832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45589 (local address 127.0.98.3:41163)
0926 13:14:12.846882 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.846885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.846933 (+ 48us) server_negotiation.cc:407] Connection header received
0926 13:14:12.847012 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.847013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.847022 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.847058 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:12.847222 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.847907 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.849475 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.850039 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.850118 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.850128 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.850372 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.850421 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:14:12.850769 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.846604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41163 (local address 127.0.98.2:45589)
0926 13:14:12.846875 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.846885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.846948 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.847064 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.847065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.847067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:12.847132 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.847133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.848061 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.848061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.849383 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.849384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.850203 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.850204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.850280 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.850284 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.850635 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.850686 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":116,"thread_start_us":125,"threads_started":1}
I20260926 13:14:12.850773 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916fcfba279420f950d690417d25cda" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" is_pre_election: true
I20260926 13:14:12.850947 2285 raft_consensus.cc:2454] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 0.
I20260926 13:14:12.851104 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916fcfba279420f950d690417d25cda" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc90a1d05c4c409b29763059c5c815" is_pre_election: true
I20260926 13:14:12.851166 2358 leader_election.cc:302] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8e73d841da451aa05ce7f7da5c67f8, 57db91d613c04eb0b5bfdcd028da811a; no voters:
I20260926 13:14:12.851274 2565 raft_consensus.cc:2454] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 0.
I20260926 13:14:12.851337 2642 raft_consensus.cc:2781] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:12.851400 2642 raft_consensus.cc:482] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:12.851424 2642 raft_consensus.cc:3037] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:12.852334 2642 raft_consensus.cc:504] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.852487 2642 leader_election.cc:288] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 election: Requested vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163)
I20260926 13:14:12.852619 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916fcfba279420f950d690417d25cda" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
I20260926 13:14:12.852689 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1916fcfba279420f950d690417d25cda" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc90a1d05c4c409b29763059c5c815"
I20260926 13:14:12.852717 2285 raft_consensus.cc:3037] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:12.852767 2565 raft_consensus.cc:3037] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:12.853440 2565 raft_consensus.cc:2454] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 1.
I20260926 13:14:12.853554 2285 raft_consensus.cc:2454] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 1.
I20260926 13:14:12.853639 2357 leader_election.cc:302] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57db91d613c04eb0b5bfdcd028da811a, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:12.853734 2642 raft_consensus.cc:2781] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:12.853870 2642 raft_consensus.cc:686] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 1 LEADER]: Becoming Leader. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Running, Role: LEADER
I20260926 13:14:12.853967 2642 consensus_queue.cc:237] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.854835 2127 catalog_manager.cc:5495] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a reported cstate change: term changed from 0 to 1, leader changed from <none> to 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "57db91d613c04eb0b5bfdcd028da811a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } health_report { overall_health: UNKNOWN } } }
W20260926 13:14:12.871170 2332 tablet.cc:2369] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:14:12 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:12.905747 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.898896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39625 (local address 127.0.0.1:52226)
0926 13:14:12.899061 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:12.899072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:12.899139 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:12.899263 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:12.899264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:12.899670 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:12.899719 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.899721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.900350 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.900351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.900871 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:12.900872 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.901210 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.901211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:12.901249 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.901810 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:12.901814 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:12.903305 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:12.904794 (+ 1489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.904801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.904809 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.905005 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.905182 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:12.905182 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:12.905182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:12.905235 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:12.905461 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:12.905480 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:12.905519 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:14:12.905571 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:14:12.905656 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260926 13:14:12.905798 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:12.898968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52226 (local address 127.0.98.62:39625)
0926 13:14:12.899078 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:12.899080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:12.899137 (+ 57us) server_negotiation.cc:407] Connection header received
0926 13:14:12.899182 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:12.899183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:12.899192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:12.899245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:12.899792 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.900269 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.900952 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:12.901140 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:12.901196 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:12.903397 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:12.903398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:12.903398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:12.903405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:12.904709 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.905073 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.905074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.905074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.905124 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:12.905287 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:12.905287 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:12.905287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:12.905407 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:12.905458 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:12.905610 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 13:14:12.905702 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":76}
I20260926 13:14:12.908025 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52226:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:14:12.908205 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:12.909668 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 2556b3dd170e442cb927014dd2d49346 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=374b2215d7a640f09edafe4b5bdf3f69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:14:12.909701 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 2556b3dd170e442cb927014dd2d49346 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=374b2215d7a640f09edafe4b5bdf3f69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:14:12.909698 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 2556b3dd170e442cb927014dd2d49346 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=374b2215d7a640f09edafe4b5bdf3f69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:14:12.909839 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556b3dd170e442cb927014dd2d49346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.909843 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556b3dd170e442cb927014dd2d49346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.909843 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2556b3dd170e442cb927014dd2d49346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:12.911154 2634 tablet_bootstrap.cc:492] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: Bootstrap starting.
I20260926 13:14:12.911301 2636 tablet_bootstrap.cc:492] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: Bootstrap starting.
I20260926 13:14:12.911325 2635 tablet_bootstrap.cc:492] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: Bootstrap starting.
I20260926 13:14:12.911550 2634 tablet_bootstrap.cc:654] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.911738 2636 tablet_bootstrap.cc:654] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.911746 2635 tablet_bootstrap.cc:654] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:12.912210 2634 tablet_bootstrap.cc:492] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: No bootstrap required, opened a new log
I20260926 13:14:12.912267 2634 ts_tablet_manager.cc:1397] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:12.912371 2635 tablet_bootstrap.cc:492] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: No bootstrap required, opened a new log
I20260926 13:14:12.912417 2635 ts_tablet_manager.cc:1397] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:12.912406 2634 raft_consensus.cc:348] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.912494 2634 raft_consensus.cc:374] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.912525 2634 raft_consensus.cc:729] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8e73d841da451aa05ce7f7da5c67f8, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.912554 2635 raft_consensus.cc:348] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.912577 2634 consensus_queue.cc:260] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.912627 2635 raft_consensus.cc:374] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.912653 2635 raft_consensus.cc:729] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.912698 2634 ts_tablet_manager.cc:1428] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:12.912696 2635 consensus_queue.cc:260] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.912782 2635 ts_tablet_manager.cc:1428] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:12.912747 2640 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:12.912848 2640 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:12.912904 2642 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:12.912990 2642 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:12.913141 2636 tablet_bootstrap.cc:492] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: No bootstrap required, opened a new log
I20260926 13:14:12.913208 2636 ts_tablet_manager.cc:1397] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260926 13:14:12.913353 2636 raft_consensus.cc:348] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.913427 2636 raft_consensus.cc:374] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:12.913450 2636 raft_consensus.cc:729] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Initialized, Role: FOLLOWER
I20260926 13:14:12.913507 2636 consensus_queue.cc:260] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:12.913594 2636 ts_tablet_manager.cc:1428] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:12.913655 2641 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:12.913744 2641 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 13:14:12.948725 2612 tablet.cc:2369] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:14:13.052951 2641 raft_consensus.cc:482] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:13.053071 2641 raft_consensus.cc:504] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.053500 2641 leader_election.cc:288] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:13.057240 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.053804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52427 (local address 127.0.98.2:44277)
0926 13:14:13.053886 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.053888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.054144 (+ 256us) server_negotiation.cc:407] Connection header received
0926 13:14:13.054214 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.054215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.054226 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.054281 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:13.054423 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.055105 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.056452 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.056814 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.056867 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.056876 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.057132 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.057173 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 13:14:13.057236 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.053696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32907 (local address 127.0.98.3:36409)
0926 13:14:13.053764 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:13.053778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:13.053869 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:13.054035 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:13.054036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:13.054038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:13.054116 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.054118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.055173 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.055176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.056254 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.056257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.056933 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.056934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.057020 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.057022 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:14:13.057067 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:14:13.057125 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":74}
I20260926 13:14:13.057236 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.053804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36409 (local address 127.0.98.1:32907)
0926 13:14:13.053872 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.053875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.053911 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:14:13.053944 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.053945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.053956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.054011 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:13.054203 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.055029 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.056346 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.056818 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.056926 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.056938 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.057098 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.057150 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 13:14:13.057564 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.053696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44277 (local address 127.0.98.3:52427)
0926 13:14:13.054052 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:13.054062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:13.054123 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:13.054310 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:13.054310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:13.054312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:13.054355 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.054356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.055203 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.055204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.056350 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.056354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.056924 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.056926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.057026 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.057031 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:13.057120 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:14:13.057161 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":137,"thread_start_us":129,"threads_started":1}
I20260926 13:14:13.057690 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556b3dd170e442cb927014dd2d49346" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" is_pre_election: true
I20260926 13:14:13.057811 2285 raft_consensus.cc:2454] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 0.
I20260926 13:14:13.057998 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556b3dd170e442cb927014dd2d49346" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57db91d613c04eb0b5bfdcd028da811a" is_pre_election: true
I20260926 13:14:13.058017 2498 leader_election.cc:302] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8e73d841da451aa05ce7f7da5c67f8, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:13.058089 2425 raft_consensus.cc:2454] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 0.
I20260926 13:14:13.058208 2641 raft_consensus.cc:2781] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:13.058265 2641 raft_consensus.cc:482] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:13.058317 2641 raft_consensus.cc:3037] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:13.059113 2641 raft_consensus.cc:504] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.059295 2641 leader_election.cc:288] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:13.059455 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556b3dd170e442cb927014dd2d49346" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
I20260926 13:14:13.059458 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2556b3dd170e442cb927014dd2d49346" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57db91d613c04eb0b5bfdcd028da811a"
I20260926 13:14:13.059544 2425 raft_consensus.cc:3037] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:13.059544 2285 raft_consensus.cc:3037] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:13.060437 2425 raft_consensus.cc:2454] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 1.
I20260926 13:14:13.060437 2285 raft_consensus.cc:2454] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 1.
I20260926 13:14:13.060632 2496 leader_election.cc:302] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57db91d613c04eb0b5bfdcd028da811a, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:13.060736 2641 raft_consensus.cc:2781] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:13.060882 2641 raft_consensus.cc:686] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [term 1 LEADER]: Becoming Leader. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Running, Role: LEADER
I20260926 13:14:13.060987 2641 consensus_queue.cc:237] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.061412 2656 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: TxnStatusTablet state changed. Reason: New leader 86bc90a1d05c4c409b29763059c5c815. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.061530 2656 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:14:13.061754 2658 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:14:13.061868 2658 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:14:13.061797 2127 catalog_manager.cc:5495] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:13.096652 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.094145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60956 (local address 127.0.98.3:41163)
0926 13:14:13.094224 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.094226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.094237 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:13.094349 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.094350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.094375 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.094446 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:13.094591 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.095076 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.095840 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.096042 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.096119 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.096317 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.096415 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.096506 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.096547 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.096567 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:14:13.096660 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.094058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41163 (local address 127.0.0.1:60956)
0926 13:14:13.094131 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:13.094150 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:13.094244 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:13.094452 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:13.094452 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:13.094454 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:13.094519 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.094522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.095166 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.095167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.095749 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.095751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.096116 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.096117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.096189 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.096202 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.096487 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.096496 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 13:14:13.096543 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:14:13.096592 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:14:13.098814 2425 raft_consensus.cc:1261] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Refusing update from remote peer 86bc90a1d05c4c409b29763059c5c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:13.098845 2285 raft_consensus.cc:1261] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc90a1d05c4c409b29763059c5c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:13.099130 2641 consensus_queue.cc:1035] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:13.099362 2641 consensus_queue.cc:1035] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:13.100303 2642 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: TxnStatusTablet state changed. Reason: New leader 86bc90a1d05c4c409b29763059c5c815. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.100394 2642 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:13.100523 2640 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: TxnStatusTablet state changed. Reason: New leader 86bc90a1d05c4c409b29763059c5c815. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.100621 2640 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:13.101372 2641 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.101460 2641 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:14:13.101559 2641 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.101625 2641 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 86bc90a1d05c4c409b29763059c5c815: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:14:13.102094 2663 mvcc.cc:205] Tried to move back new op lower bound from 7333594943889588224 to 7333594943738167296. Current Snapshot: MvccSnapshot[applied={T|T < 7333594943889588224}]
I20260926 13:14:13.102414 2642 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.102496 2642 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 57db91d613c04eb0b5bfdcd028da811a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:13.102792 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333594943889588224 to 7333594943738167296. Current Snapshot: MvccSnapshot[applied={T|T < 7333594943889588224}]
I20260926 13:14:13.103060 2640 tablet_replica.cc:439] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } } }
I20260926 13:14:13.103153 2640 tablet_replica.cc:442] T 2556b3dd170e442cb927014dd2d49346 P 3b8e73d841da451aa05ce7f7da5c67f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:14:13.103538 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333594943889588224 to 7333594943738167296. Current Snapshot: MvccSnapshot[applied={T|T < 7333594943889588224 or (T in {7333594943889588224})}]
I20260926 13:14:13.106758 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.103879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.0.98.2:44277)
0926 13:14:13.103923 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.103925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.103934 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:13.104004 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.104005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.104024 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.104095 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:13.104174 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.104935 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.105672 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.106010 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.106078 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.106308 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.106485 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.106570 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.106659 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.106685 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:14:13.111832 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.109347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51681 (local address 127.0.98.3:41163)
0926 13:14:13.109396 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.109398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.109497 (+ 99us) server_negotiation.cc:407] Connection header received
0926 13:14:13.109568 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.109569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.109580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.109636 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:13.109779 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.110324 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.111097 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.111302 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.111385 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.111580 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.111614 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.111684 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.111738 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.111757 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:14:13.111861 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.109261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41163 (local address 127.0.98.2:51681)
0926 13:14:13.109414 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:13.109425 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:13.109514 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:13.109639 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:13.109640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:13.109641 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:13.109700 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.109702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.110401 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.110402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.111005 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.111007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.111387 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.111388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.111453 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.111473 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.111685 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.111687 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:14:13.111735 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:14:13.111779 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260926 13:14:13.114570 2285 raft_consensus.cc:1261] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Refusing update from remote peer 57db91d613c04eb0b5bfdcd028da811a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:13.114619 2565 raft_consensus.cc:1261] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Refusing update from remote peer 57db91d613c04eb0b5bfdcd028da811a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:13.114867 2642 consensus_queue.cc:1035] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:13.115015 2644 consensus_queue.cc:1035] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:13.123968 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.121491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36571 (local address 127.0.98.2:44277)
0926 13:14:13.121548 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.121550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.121559 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:13.121582 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.121583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.121593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.121650 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:13.121746 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.122276 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.123358 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.123560 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.123617 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.123780 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.123805 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.123841 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.123893 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.123911 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:14:13.124010 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.121422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44277 (local address 127.0.98.3:36571)
0926 13:14:13.121491 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:13.121503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:13.121537 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:13.121650 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:13.121651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:13.121652 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:13.121698 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.121699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.122346 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.122347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.123275 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.123278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.123631 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.123632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.123686 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.123700 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.123845 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.123846 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:14:13.123891 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:14:13.123937 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:14:13.133093 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:13.133216 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:13.133980 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Sending DeleteTablet for 3 replicas of tablet 1916fcfba279420f950d690417d25cda
I20260926 13:14:13.134341 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916fcfba279420f950d690417d25cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33560
I20260926 13:14:13.134594 2679 tablet_replica.cc:330] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: stopping tablet replica
I20260926 13:14:13.134568 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916fcfba279420f950d690417d25cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60946
I20260926 13:14:13.134789 2679 raft_consensus.cc:2229] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:13.134938 2679 raft_consensus.cc:2258] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:13.135043 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1916fcfba279420f950d690417d25cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47136
I20260926 13:14:13.135270 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:13.135447 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:13.135730 2681 tablet_replica.cc:330] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: stopping tablet replica
I20260926 13:14:13.135900 2681 raft_consensus.cc:2229] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:13.136040 2681 raft_consensus.cc:2258] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:13.136596 2681 ts_tablet_manager.cc:1905] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:13.136921 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 30a3aaee75c541cd8e7f11393aea4ba2 (DEFAULT_TABLE table=test-table [id=c2b9d980de7c4569acfb5ad76847e42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:13.137033 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a3aaee75c541cd8e7f11393aea4ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:13.137125 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 30a3aaee75c541cd8e7f11393aea4ba2 (DEFAULT_TABLE table=test-table [id=c2b9d980de7c4569acfb5ad76847e42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:13.137234 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a3aaee75c541cd8e7f11393aea4ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:13.138492 2635 tablet_bootstrap.cc:492] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Bootstrap starting.
I20260926 13:14:13.138608 2636 tablet_bootstrap.cc:492] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Bootstrap starting.
I20260926 13:14:13.137043 2679 ts_tablet_manager.cc:1905] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:13.138908 2635 tablet_bootstrap.cc:654] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:13.138926 2681 ts_tablet_manager.cc:1918] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:13.139020 2636 tablet_bootstrap.cc:654] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:13.139137 2681 log.cc:1196] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/1916fcfba279420f950d690417d25cda
I20260926 13:14:13.139470 2681 ts_tablet_manager.cc:1939] T 1916fcfba279420f950d690417d25cda P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting consensus metadata
I20260926 13:14:13.139734 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 30a3aaee75c541cd8e7f11393aea4ba2 (DEFAULT_TABLE table=test-table [id=c2b9d980de7c4569acfb5ad76847e42b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:13.139832 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30a3aaee75c541cd8e7f11393aea4ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:13.140209 2679 ts_tablet_manager.cc:1918] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:13.140280 2679 log.cc:1196] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/1916fcfba279420f950d690417d25cda
I20260926 13:14:13.140554 2679 ts_tablet_manager.cc:1939] T 1916fcfba279420f950d690417d25cda P 57db91d613c04eb0b5bfdcd028da811a: Deleting consensus metadata
I20260926 13:14:13.140851 2634 tablet_bootstrap.cc:492] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Bootstrap starting.
I20260926 13:14:13.140924 2112 catalog_manager.cc:4841] TS 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277): tablet 1916fcfba279420f950d690417d25cda (table test-table [id=63bc9cf16b4a4e6f8f5e023c65678bc9]) successfully deleted
I20260926 13:14:13.140950 2114 catalog_manager.cc:4841] TS 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907): tablet 1916fcfba279420f950d690417d25cda (table test-table [id=63bc9cf16b4a4e6f8f5e023c65678bc9]) successfully deleted
I20260926 13:14:13.141253 2634 tablet_bootstrap.cc:654] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:13.141445 2636 tablet_bootstrap.cc:492] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: No bootstrap required, opened a new log
I20260926 13:14:13.141501 2636 ts_tablet_manager.cc:1397] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260926 13:14:13.141654 2636 raft_consensus.cc:348] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.141743 2636 raft_consensus.cc:374] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:13.141772 2636 raft_consensus.cc:729] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Initialized, Role: FOLLOWER
I20260926 13:14:13.141813 2636 consensus_queue.cc:260] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.141917 2636 ts_tablet_manager.cc:1428] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:13.142185 2634 tablet_bootstrap.cc:492] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: No bootstrap required, opened a new log
I20260926 13:14:13.142233 2634 ts_tablet_manager.cc:1397] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:13.142354 2634 raft_consensus.cc:348] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.142448 2634 raft_consensus.cc:374] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:13.142473 2634 raft_consensus.cc:729] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8e73d841da451aa05ce7f7da5c67f8, State: Initialized, Role: FOLLOWER
I20260926 13:14:13.142514 2634 consensus_queue.cc:260] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.142607 2634 ts_tablet_manager.cc:1428] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:13.143133 2680 tablet_replica.cc:330] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: stopping tablet replica
I20260926 13:14:13.143247 2680 raft_consensus.cc:2229] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:13.143309 2680 raft_consensus.cc:2258] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:13.143697 2680 ts_tablet_manager.cc:1905] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:13.145104 2680 ts_tablet_manager.cc:1918] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:13.145192 2680 log.cc:1196] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/1916fcfba279420f950d690417d25cda
I20260926 13:14:13.145434 2680 ts_tablet_manager.cc:1939] T 1916fcfba279420f950d690417d25cda P 86bc90a1d05c4c409b29763059c5c815: Deleting consensus metadata
I20260926 13:14:13.145819 2113 catalog_manager.cc:4841] TS 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163): tablet 1916fcfba279420f950d690417d25cda (table test-table [id=63bc9cf16b4a4e6f8f5e023c65678bc9]) successfully deleted
I20260926 13:14:13.146643 2635 tablet_bootstrap.cc:492] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: No bootstrap required, opened a new log
I20260926 13:14:13.146713 2635 ts_tablet_manager.cc:1397] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260926 13:14:13.146867 2635 raft_consensus.cc:348] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.146976 2635 raft_consensus.cc:374] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:13.147008 2635 raft_consensus.cc:729] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Initialized, Role: FOLLOWER
I20260926 13:14:13.147050 2635 consensus_queue.cc:260] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.147146 2635 ts_tablet_manager.cc:1428] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:13.265338 2673 raft_consensus.cc:482] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:13.265470 2673 raft_consensus.cc:504] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.265647 2673 leader_election.cc:288] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:13.265864 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a3aaee75c541cd8e7f11393aea4ba2" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" is_pre_election: true
I20260926 13:14:13.265875 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a3aaee75c541cd8e7f11393aea4ba2" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57db91d613c04eb0b5bfdcd028da811a" is_pre_election: true
I20260926 13:14:13.265959 2285 raft_consensus.cc:2454] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 0.
I20260926 13:14:13.265985 2425 raft_consensus.cc:2454] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 0.
I20260926 13:14:13.266115 2498 leader_election.cc:302] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8e73d841da451aa05ce7f7da5c67f8, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:13.266230 2673 raft_consensus.cc:2781] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:13.266268 2673 raft_consensus.cc:482] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:13.266301 2673 raft_consensus.cc:3037] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:13.267145 2673 raft_consensus.cc:504] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.267325 2673 leader_election.cc:288] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:13.267493 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a3aaee75c541cd8e7f11393aea4ba2" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
I20260926 13:14:13.267570 2285 raft_consensus.cc:3037] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:13.267553 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30a3aaee75c541cd8e7f11393aea4ba2" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57db91d613c04eb0b5bfdcd028da811a"
I20260926 13:14:13.267620 2425 raft_consensus.cc:3037] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:13.268369 2425 raft_consensus.cc:2454] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 1.
I20260926 13:14:13.268373 2285 raft_consensus.cc:2454] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 1.
I20260926 13:14:13.268536 2498 leader_election.cc:302] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b8e73d841da451aa05ce7f7da5c67f8, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:13.268628 2673 raft_consensus.cc:2781] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:13.268672 2673 raft_consensus.cc:686] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 1 LEADER]: Becoming Leader. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Running, Role: LEADER
I20260926 13:14:13.268718 2673 consensus_queue.cc:237] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:13.269340 2127 catalog_manager.cc:5495] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:13.292517 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.290263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.0.98.3:41163)
0926 13:14:13.290311 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.290313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.290320 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:13.290345 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.290346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.290358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.290400 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:13.290547 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.291036 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.291877 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.292065 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.292130 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.292252 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.292279 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.292317 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.292375 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.292394 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:14:13.296957 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.294676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41163 (local address 127.0.98.3:42677)
0926 13:14:13.294841 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:13.294853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:13.294910 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:13.295039 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:13.295039 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:13.295041 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:14:13.295077 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.295078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.295704 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.295704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.296228 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:13.296229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.296587 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.296587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:13.296634 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.296644 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.296810 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.296812 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:14:13.296857 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:14:13.296895 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 13:14:13.296958 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:13.294849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42677 (local address 127.0.98.3:41163)
0926 13:14:13.294888 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:13.294890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:13.294898 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:14:13.294974 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:13.294974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:13.294982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:13.295036 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:13.295149 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.295610 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.296325 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:13.296506 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:13.296587 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:13.296718 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:13.296747 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:13.296795 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:13.296861 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 13:14:13.296881 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:13.299216 2285 raft_consensus.cc:1261] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc90a1d05c4c409b29763059c5c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:13.299218 2425 raft_consensus.cc:1261] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Refusing update from remote peer 86bc90a1d05c4c409b29763059c5c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:13.299432 2641 consensus_queue.cc:1035] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:13.299556 2656 consensus_queue.cc:1035] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:14:13.303773 2661 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:14:14.823518 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:14.823637 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:14.824457 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Sending DeleteTablet for 3 replicas of tablet 30a3aaee75c541cd8e7f11393aea4ba2
I20260926 13:14:14.824725 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a3aaee75c541cd8e7f11393aea4ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33560
I20260926 13:14:14.824761 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a3aaee75c541cd8e7f11393aea4ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47136
I20260926 13:14:14.824757 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 30a3aaee75c541cd8e7f11393aea4ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60946
I20260926 13:14:14.824940 2710 tablet_replica.cc:330] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: stopping tablet replica
I20260926 13:14:14.825021 2710 raft_consensus.cc:2229] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:14.825120 2710 raft_consensus.cc:2258] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:14.825222 2709 tablet_replica.cc:330] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: stopping tablet replica
I20260926 13:14:14.825304 2709 raft_consensus.cc:2229] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:14.825345 2709 raft_consensus.cc:2258] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:14.827260 2710 ts_tablet_manager.cc:1905] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:14.827478 2709 ts_tablet_manager.cc:1905] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:14.828588 2708 tablet_replica.cc:330] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: stopping tablet replica
I20260926 13:14:14.828679 2708 raft_consensus.cc:2229] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:14.828727 2708 raft_consensus.cc:2258] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:14.828948 2708 ts_tablet_manager.cc:1905] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:14.829037 2710 ts_tablet_manager.cc:1918] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:14:14.829101 2710 log.cc:1196] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/30a3aaee75c541cd8e7f11393aea4ba2
I20260926 13:14:14.829345 2709 ts_tablet_manager.cc:1918] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:14:14.829391 2710 ts_tablet_manager.cc:1939] T 30a3aaee75c541cd8e7f11393aea4ba2 P 86bc90a1d05c4c409b29763059c5c815: Deleting consensus metadata
I20260926 13:14:14.829412 2709 log.cc:1196] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/30a3aaee75c541cd8e7f11393aea4ba2
I20260926 13:14:14.829627 2709 ts_tablet_manager.cc:1939] T 30a3aaee75c541cd8e7f11393aea4ba2 P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting consensus metadata
I20260926 13:14:14.829739 2113 catalog_manager.cc:4841] TS 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163): tablet 30a3aaee75c541cd8e7f11393aea4ba2 (table test-table [id=c2b9d980de7c4569acfb5ad76847e42b]) successfully deleted
I20260926 13:14:14.829914 2114 catalog_manager.cc:4841] TS 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907): tablet 30a3aaee75c541cd8e7f11393aea4ba2 (table test-table [id=c2b9d980de7c4569acfb5ad76847e42b]) successfully deleted
I20260926 13:14:14.830475 2708 ts_tablet_manager.cc:1918] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:14:14.830549 2708 log.cc:1196] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/30a3aaee75c541cd8e7f11393aea4ba2
I20260926 13:14:14.830753 2708 ts_tablet_manager.cc:1939] T 30a3aaee75c541cd8e7f11393aea4ba2 P 57db91d613c04eb0b5bfdcd028da811a: Deleting consensus metadata
I20260926 13:14:14.831225 2112 catalog_manager.cc:4841] TS 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277): tablet 30a3aaee75c541cd8e7f11393aea4ba2 (table test-table [id=c2b9d980de7c4569acfb5ad76847e42b]) successfully deleted
Sep 26 13:14:14 dist-test-slave-f864 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:14:14 dist-test-slave-f864 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428454, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:14.855028 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:14.842846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52240 (local address 127.0.98.62:39625)
0926 13:14:14.842990 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:14.842991 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:14.843028 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:14:14.843098 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:14.843099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:14.843110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:14.843170 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:14.843704 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:14.844205 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:14.844892 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:14.845084 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:14.845140 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:14.852287 (+ 7147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:14.852289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:14.852289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:14.852298 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:14.853848 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:14.854343 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:14.854344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:14.854344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:14.854403 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:14.854577 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:14.854578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:14.854578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:14.854704 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:14.854794 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:14.854880 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:14:14.854951 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260926 13:14:14.856622 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:14.856804 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:14.858253 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 76712e04bfea4619ad1d020c7b721e38 (DEFAULT_TABLE table=test-table [id=b48e91fc28314e60b3cdcbe738d02d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:14.858261 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 76712e04bfea4619ad1d020c7b721e38 (DEFAULT_TABLE table=test-table [id=b48e91fc28314e60b3cdcbe738d02d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:14.858290 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 76712e04bfea4619ad1d020c7b721e38 (DEFAULT_TABLE table=test-table [id=b48e91fc28314e60b3cdcbe738d02d2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:14.858420 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76712e04bfea4619ad1d020c7b721e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:14.858422 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76712e04bfea4619ad1d020c7b721e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:14.858423 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76712e04bfea4619ad1d020c7b721e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:14.859651 2719 tablet_bootstrap.cc:492] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Bootstrap starting.
I20260926 13:14:14.859850 2720 tablet_bootstrap.cc:492] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Bootstrap starting.
I20260926 13:14:14.859859 2721 tablet_bootstrap.cc:492] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Bootstrap starting.
I20260926 13:14:14.860069 2719 tablet_bootstrap.cc:654] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:14.860246 2721 tablet_bootstrap.cc:654] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:14.860263 2720 tablet_bootstrap.cc:654] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:14.860744 2719 tablet_bootstrap.cc:492] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: No bootstrap required, opened a new log
I20260926 13:14:14.860800 2719 ts_tablet_manager.cc:1397] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:14.860850 2720 tablet_bootstrap.cc:492] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: No bootstrap required, opened a new log
I20260926 13:14:14.860903 2720 ts_tablet_manager.cc:1397] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:14.860966 2719 raft_consensus.cc:348] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.861055 2719 raft_consensus.cc:374] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:14.861023 2720 raft_consensus.cc:348] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.861078 2719 raft_consensus.cc:729] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8e73d841da451aa05ce7f7da5c67f8, State: Initialized, Role: FOLLOWER
I20260926 13:14:14.861086 2720 raft_consensus.cc:374] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:14.861128 2720 raft_consensus.cc:729] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Initialized, Role: FOLLOWER
I20260926 13:14:14.861135 2719 consensus_queue.cc:260] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.861173 2720 consensus_queue.cc:260] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.861317 2719 ts_tablet_manager.cc:1428] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:14.861333 2720 ts_tablet_manager.cc:1428] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:14.861547 2721 tablet_bootstrap.cc:492] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: No bootstrap required, opened a new log
I20260926 13:14:14.861604 2721 ts_tablet_manager.cc:1397] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:14.861711 2721 raft_consensus.cc:348] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.861802 2721 raft_consensus.cc:374] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:14.861829 2721 raft_consensus.cc:729] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Initialized, Role: FOLLOWER
I20260926 13:14:14.861876 2721 consensus_queue.cc:260] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.861981 2721 ts_tablet_manager.cc:1428] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:14.932507 2696 raft_consensus.cc:482] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:14.932632 2696 raft_consensus.cc:504] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.932827 2696 leader_election.cc:288] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:14.933033 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76712e04bfea4619ad1d020c7b721e38" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57db91d613c04eb0b5bfdcd028da811a" is_pre_election: true
I20260926 13:14:14.933034 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76712e04bfea4619ad1d020c7b721e38" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" is_pre_election: true
I20260926 13:14:14.933149 2425 raft_consensus.cc:2454] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 0.
I20260926 13:14:14.933149 2284 raft_consensus.cc:2454] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 0.
I20260926 13:14:14.933346 2496 leader_election.cc:302] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57db91d613c04eb0b5bfdcd028da811a, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:14.933476 2696 raft_consensus.cc:2781] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:14.933555 2696 raft_consensus.cc:482] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:14.933573 2696 raft_consensus.cc:3037] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:14.934297 2696 raft_consensus.cc:504] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.934449 2696 leader_election.cc:288] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 election: Requested vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277)
I20260926 13:14:14.934600 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76712e04bfea4619ad1d020c7b721e38" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
I20260926 13:14:14.934624 2425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76712e04bfea4619ad1d020c7b721e38" candidate_uuid: "86bc90a1d05c4c409b29763059c5c815" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57db91d613c04eb0b5bfdcd028da811a"
I20260926 13:14:14.934701 2425 raft_consensus.cc:3037] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:14.934702 2284 raft_consensus.cc:3037] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:14.935515 2425 raft_consensus.cc:2454] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 1.
I20260926 13:14:14.935515 2284 raft_consensus.cc:2454] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86bc90a1d05c4c409b29763059c5c815 in term 1.
I20260926 13:14:14.935771 2496 leader_election.cc:302] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b8e73d841da451aa05ce7f7da5c67f8, 57db91d613c04eb0b5bfdcd028da811a, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:14.935882 2696 raft_consensus.cc:2781] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:14.935937 2696 raft_consensus.cc:686] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 1 LEADER]: Becoming Leader. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Running, Role: LEADER
I20260926 13:14:14.936036 2696 consensus_queue.cc:237] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.936718 2127 catalog_manager.cc:5495] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "86bc90a1d05c4c409b29763059c5c815" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:14.957823 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:14.955413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.0.98.3:41163)
0926 13:14:14.955576 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:14.955578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:14.955585 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:14.955605 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:14.955605 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:14.955615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:14.955684 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:14.955831 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:14.956324 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:14.957097 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:14.957315 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:14.957399 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:14.957550 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:14.957585 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:14.957654 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:14.957707 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 13:14:14.957729 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260926 13:14:14.959914 2425 raft_consensus.cc:1261] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Refusing update from remote peer 86bc90a1d05c4c409b29763059c5c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:14.959918 2284 raft_consensus.cc:1261] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Refusing update from remote peer 86bc90a1d05c4c409b29763059c5c815: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:14.960158 2691 consensus_queue.cc:1035] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:14.960263 2696 consensus_queue.cc:1035] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:14.970355 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:14.970466 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:14.971143 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Sending DeleteTablet for 3 replicas of tablet 76712e04bfea4619ad1d020c7b721e38
I20260926 13:14:14.971441 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 76712e04bfea4619ad1d020c7b721e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47136
I20260926 13:14:14.971462 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 76712e04bfea4619ad1d020c7b721e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33560
I20260926 13:14:14.971547 2709 tablet_replica.cc:330] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: stopping tablet replica
I20260926 13:14:14.971553 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 76712e04bfea4619ad1d020c7b721e38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60946
I20260926 13:14:14.971573 2708 tablet_replica.cc:330] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: stopping tablet replica
I20260926 13:14:14.971603 2709 raft_consensus.cc:2229] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:14.971623 2708 raft_consensus.cc:2229] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:14.971669 2708 raft_consensus.cc:2258] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:14.971673 2709 raft_consensus.cc:2258] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:14.971750 2710 tablet_replica.cc:330] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: stopping tablet replica
I20260926 13:14:14.971807 2710 raft_consensus.cc:2229] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:14.971853 2710 raft_consensus.cc:2258] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:14.971784 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:14.971925 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:14.972078 2709 ts_tablet_manager.cc:1905] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:14.972087 2710 ts_tablet_manager.cc:1905] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:14.972491 2708 ts_tablet_manager.cc:1905] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:14.973353 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 333a3880f7174f288dae83ec0bcb1c51 (DEFAULT_TABLE table=test-table [id=0a7718c2f46346cfaac33ed83b8c7427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:14.973453 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333a3880f7174f288dae83ec0bcb1c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:14.973708 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 333a3880f7174f288dae83ec0bcb1c51 (DEFAULT_TABLE table=test-table [id=0a7718c2f46346cfaac33ed83b8c7427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:14.973805 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333a3880f7174f288dae83ec0bcb1c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:14.973825 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 333a3880f7174f288dae83ec0bcb1c51 (DEFAULT_TABLE table=test-table [id=0a7718c2f46346cfaac33ed83b8c7427]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:14.973886 2709 ts_tablet_manager.cc:1918] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:14.973922 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 333a3880f7174f288dae83ec0bcb1c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:14.973937 2709 log.cc:1196] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/76712e04bfea4619ad1d020c7b721e38
I20260926 13:14:14.974154 2709 ts_tablet_manager.cc:1939] T 76712e04bfea4619ad1d020c7b721e38 P 3b8e73d841da451aa05ce7f7da5c67f8: Deleting consensus metadata
I20260926 13:14:14.974311 2708 ts_tablet_manager.cc:1918] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:14.974359 2708 log.cc:1196] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/76712e04bfea4619ad1d020c7b721e38
I20260926 13:14:14.974488 2114 catalog_manager.cc:4841] TS 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907): tablet 76712e04bfea4619ad1d020c7b721e38 (table test-table [id=b48e91fc28314e60b3cdcbe738d02d2a]) successfully deleted
I20260926 13:14:14.974567 2708 ts_tablet_manager.cc:1939] T 76712e04bfea4619ad1d020c7b721e38 P 57db91d613c04eb0b5bfdcd028da811a: Deleting consensus metadata
I20260926 13:14:14.975020 2710 ts_tablet_manager.cc:1918] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:14:14.975059 2720 tablet_bootstrap.cc:492] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Bootstrap starting.
I20260926 13:14:14.975107 2710 log.cc:1196] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/76712e04bfea4619ad1d020c7b721e38
I20260926 13:14:14.975175 2719 tablet_bootstrap.cc:492] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8: Bootstrap starting.
I20260926 13:14:14.975322 2710 ts_tablet_manager.cc:1939] T 76712e04bfea4619ad1d020c7b721e38 P 86bc90a1d05c4c409b29763059c5c815: Deleting consensus metadata
I20260926 13:14:14.975343 2720 tablet_bootstrap.cc:654] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:14.975602 2719 tablet_bootstrap.cc:654] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:14.975826 2720 tablet_bootstrap.cc:492] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: No bootstrap required, opened a new log
I20260926 13:14:14.975891 2720 ts_tablet_manager.cc:1397] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:14:14.975906 2113 catalog_manager.cc:4841] TS 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163): tablet 76712e04bfea4619ad1d020c7b721e38 (table test-table [id=b48e91fc28314e60b3cdcbe738d02d2a]) successfully deleted
I20260926 13:14:14.976006 2720 raft_consensus.cc:348] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.976082 2720 raft_consensus.cc:374] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:14.976104 2720 raft_consensus.cc:729] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Initialized, Role: FOLLOWER
I20260926 13:14:14.976148 2720 consensus_queue.cc:260] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.976253 2720 ts_tablet_manager.cc:1428] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:14.976392 2719 tablet_bootstrap.cc:492] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8: No bootstrap required, opened a new log
I20260926 13:14:14.976441 2719 ts_tablet_manager.cc:1397] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:14.976594 2719 raft_consensus.cc:348] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.976670 2719 raft_consensus.cc:374] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:14.976698 2719 raft_consensus.cc:729] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b8e73d841da451aa05ce7f7da5c67f8, State: Initialized, Role: FOLLOWER
I20260926 13:14:14.976734 2719 consensus_queue.cc:260] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.976825 2719 ts_tablet_manager.cc:1428] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:14.977021 2721 tablet_bootstrap.cc:492] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Bootstrap starting.
I20260926 13:14:14.977308 2721 tablet_bootstrap.cc:654] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:14.977797 2112 catalog_manager.cc:4841] TS 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277): tablet 76712e04bfea4619ad1d020c7b721e38 (table test-table [id=b48e91fc28314e60b3cdcbe738d02d2a]) successfully deleted
I20260926 13:14:14.977939 2721 tablet_bootstrap.cc:492] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: No bootstrap required, opened a new log
I20260926 13:14:14.977986 2721 ts_tablet_manager.cc:1397] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:14.978127 2721 raft_consensus.cc:348] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.978220 2721 raft_consensus.cc:374] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:14.978246 2721 raft_consensus.cc:729] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86bc90a1d05c4c409b29763059c5c815, State: Initialized, Role: FOLLOWER
I20260926 13:14:14.978299 2721 consensus_queue.cc:260] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.978389 2721 ts_tablet_manager.cc:1428] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:14.980962 2726 raft_consensus.cc:482] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:14.981026 2726 raft_consensus.cc:504] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.981156 2726 leader_election.cc:288] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163)
I20260926 13:14:14.981303 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a3880f7174f288dae83ec0bcb1c51" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc90a1d05c4c409b29763059c5c815" is_pre_election: true
I20260926 13:14:14.981293 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a3880f7174f288dae83ec0bcb1c51" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" is_pre_election: true
I20260926 13:14:14.981374 2565 raft_consensus.cc:2454] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 0.
I20260926 13:14:14.981374 2284 raft_consensus.cc:2454] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 0.
I20260926 13:14:14.981492 2357 leader_election.cc:302] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57db91d613c04eb0b5bfdcd028da811a, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:14.981650 2726 raft_consensus.cc:2781] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:14.981690 2726 raft_consensus.cc:482] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:14.981709 2726 raft_consensus.cc:3037] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:14.982340 2726 raft_consensus.cc:504] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.982470 2726 leader_election.cc:288] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 election: Requested vote from peers 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907), 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163)
I20260926 13:14:14.982592 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a3880f7174f288dae83ec0bcb1c51" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86bc90a1d05c4c409b29763059c5c815"
I20260926 13:14:14.982590 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "333a3880f7174f288dae83ec0bcb1c51" candidate_uuid: "57db91d613c04eb0b5bfdcd028da811a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b8e73d841da451aa05ce7f7da5c67f8"
I20260926 13:14:14.982657 2565 raft_consensus.cc:3037] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:14.982661 2284 raft_consensus.cc:3037] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:14.983280 2565 raft_consensus.cc:2454] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 1.
I20260926 13:14:14.983364 2284 raft_consensus.cc:2454] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 57db91d613c04eb0b5bfdcd028da811a in term 1.
I20260926 13:14:14.983438 2357 leader_election.cc:302] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57db91d613c04eb0b5bfdcd028da811a, 86bc90a1d05c4c409b29763059c5c815; no voters:
I20260926 13:14:14.983521 2726 raft_consensus.cc:2781] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:14.983620 2726 raft_consensus.cc:686] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 1 LEADER]: Becoming Leader. State: Replica: 57db91d613c04eb0b5bfdcd028da811a, State: Running, Role: LEADER
I20260926 13:14:14.983659 2726 consensus_queue.cc:237] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } }
I20260926 13:14:14.984222 2128 catalog_manager.cc:5495] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a reported cstate change: term changed from 0 to 1, leader changed from <none> to 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "57db91d613c04eb0b5bfdcd028da811a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "57db91d613c04eb0b5bfdcd028da811a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 } health_report { overall_health: UNKNOWN } } }
I20260926 13:14:14.992022 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:14.989987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.98.2:44277)
0926 13:14:14.990150 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:14.990152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:14.990174 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:14:14.990197 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:14.990197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:14.990208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:14.990271 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:14.990368 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:14.990839 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:14.991502 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:14.991686 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:14.991737 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:14.991844 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:14.991868 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:14.991900 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:14.991946 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 13:14:14.991962 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":119,"threads_started":1}
I20260926 13:14:14.994293 2565 raft_consensus.cc:1261] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Refusing update from remote peer 57db91d613c04eb0b5bfdcd028da811a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:14.994457 2284 raft_consensus.cc:1261] T 333a3880f7174f288dae83ec0bcb1c51 P 3b8e73d841da451aa05ce7f7da5c67f8 [term 1 FOLLOWER]: Refusing update from remote peer 57db91d613c04eb0b5bfdcd028da811a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:14.994503 2736 consensus_queue.cc:1035] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86bc90a1d05c4c409b29763059c5c815" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:14.994644 2726 consensus_queue.cc:1035] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b8e73d841da451aa05ce7f7da5c67f8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:14:14.998597 2692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 13:14:16.515600 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:16.515714 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:16.516495 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28f406e81b124600b181ce549f01e3d3: Sending DeleteTablet for 3 replicas of tablet 333a3880f7174f288dae83ec0bcb1c51
I20260926 13:14:16.516757 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 333a3880f7174f288dae83ec0bcb1c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60946
I20260926 13:14:16.516757 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet 333a3880f7174f288dae83ec0bcb1c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33560
I20260926 13:14:16.516916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 2200
I20260926 13:14:16.516937 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 333a3880f7174f288dae83ec0bcb1c51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47136
I20260926 13:14:16.517800 2765 tablet_replica.cc:330] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: stopping tablet replica
I20260926 13:14:16.517906 2765 raft_consensus.cc:2229] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:16.517989 2765 raft_consensus.cc:2258] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:16.518307 2765 ts_tablet_manager.cc:1905] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:16.518538 2766 tablet_replica.cc:330] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: stopping tablet replica
I20260926 13:14:16.518638 2766 raft_consensus.cc:2229] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:16.518749 2766 raft_consensus.cc:2258] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:16.519029 2766 ts_tablet_manager.cc:1905] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:16.520291 2765 ts_tablet_manager.cc:1918] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:14:16.520354 2765 log.cc:1196] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/333a3880f7174f288dae83ec0bcb1c51
I20260926 13:14:16.520566 2765 ts_tablet_manager.cc:1939] T 333a3880f7174f288dae83ec0bcb1c51 P 86bc90a1d05c4c409b29763059c5c815: Deleting consensus metadata
I20260926 13:14:16.520881 2766 ts_tablet_manager.cc:1918] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 13:14:16.520893 2113 catalog_manager.cc:4841] TS 86bc90a1d05c4c409b29763059c5c815 (127.0.98.3:41163): tablet 333a3880f7174f288dae83ec0bcb1c51 (table test-table [id=0a7718c2f46346cfaac33ed83b8c7427]) successfully deleted
I20260926 13:14:16.520938 2766 log.cc:1196] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/333a3880f7174f288dae83ec0bcb1c51
I20260926 13:14:16.521186 2766 ts_tablet_manager.cc:1939] T 333a3880f7174f288dae83ec0bcb1c51 P 57db91d613c04eb0b5bfdcd028da811a: Deleting consensus metadata
W20260926 13:14:16.521965 2114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32907 (error 108)
I20260926 13:14:16.522050 2112 catalog_manager.cc:4841] TS 57db91d613c04eb0b5bfdcd028da811a (127.0.98.2:44277): tablet 333a3880f7174f288dae83ec0bcb1c51 (table test-table [id=0a7718c2f46346cfaac33ed83b8c7427]) successfully deleted
I20260926 13:14:16.522363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 2340
I20260926 13:14:16.523043 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:16.522453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32907 (local address 127.0.0.1:39912)
0926 13:14:16.522956 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:16.523010 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":69,"threads_started":1}
W20260926 13:14:16.523188 2114 catalog_manager.cc:4584] TS 3b8e73d841da451aa05ce7f7da5c67f8 (127.0.98.1:32907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 333a3880f7174f288dae83ec0bcb1c51: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32907: connect: Connection refused (error 111)
I20260926 13:14:16.529134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 2480
I20260926 13:14:16.534382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 2096
2026-09-26T13:14:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5653 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:16 dist-test-slave-f864 krb5kdc[2771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:16 dist-test-slave-f864 krb5kdc[2771](info): set up 2 sockets
Sep 26 13:14:16 dist-test-slave-f864 krb5kdc[2771](info): commencing operation
krb5kdc: starting...
W20260926 13:14:17.594000 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:17 dist-test-slave-f864 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:17.721966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:17.804056 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:17.804308 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:17.804335 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:17.805601 2784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:17.805660 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:17.805675 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:17.805688 2784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:17.805701 2784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:17.807341 2784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:17.807610 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:17.807871 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:17.813385 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:17.813483 2784 server_base.cc:1034] running on GCE node
W20260926 13:14:17.813347 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:17.813347 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:17.813925 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:17.814277 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:17.815446 2784 hybrid_clock.cc:648] HybridClock initialized: now 1790428457815421 us; error 61 us; skew 500 ppm
Sep 26 13:14:17 dist-test-slave-f864 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:17.818089 2784 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:17.818590 2784 webserver.cc:460] Webserver started at http://127.0.98.62:40965/ using document root <none> and password file <none>
I20260926 13:14:17.818773 2784 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:17.818819 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:17.818910 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:17.819926 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "93399b8829a8455497933b56bb8032a6"
format_stamp: "Formatted at 2026-09-26 13:14:17 on dist-test-slave-f864"
server_key: "c193ff5d9fef43e30687b6adcbae2077"
server_key_iv: "cb6c6c14ac9bab0ab953e25ecaf97cb5"
server_key_version: "encryptionkey@0"
I20260926 13:14:17.820231 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "93399b8829a8455497933b56bb8032a6"
format_stamp: "Formatted at 2026-09-26 13:14:17 on dist-test-slave-f864"
server_key: "c193ff5d9fef43e30687b6adcbae2077"
server_key_iv: "cb6c6c14ac9bab0ab953e25ecaf97cb5"
server_key_version: "encryptionkey@0"
I20260926 13:14:17.821489 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:14:17.822207 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:17.822407 2784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:17.822503 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "93399b8829a8455497933b56bb8032a6"
format_stamp: "Formatted at 2026-09-26 13:14:17 on dist-test-slave-f864"
server_key: "c193ff5d9fef43e30687b6adcbae2077"
server_key_iv: "cb6c6c14ac9bab0ab953e25ecaf97cb5"
server_key_version: "encryptionkey@0"
I20260926 13:14:17.822575 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:17.837373 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:17.839464 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:17.839619 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:17.843684 2784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44309
I20260926 13:14:17.843775 2851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44309 every 8 connection(s)
I20260926 13:14:17.844288 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:17.845034 2852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:17.846884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 2784
I20260926 13:14:17.846953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:17.847160 392 external_mini_cluster.cc:1428] Setting key ebb9d577b5c569c92cad9c87e1840a5d
I20260926 13:14:17.847985 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6: Bootstrap starting.
I20260926 13:14:17.848694 2852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:17.849100 2852 log.cc:824] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:17.849815 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6: No bootstrap required, opened a new log
I20260926 13:14:17.851717 2852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93399b8829a8455497933b56bb8032a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44309 } }
I20260926 13:14:17.851858 2852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:17.851883 2852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93399b8829a8455497933b56bb8032a6, State: Initialized, Role: FOLLOWER
I20260926 13:14:17.852028 2852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93399b8829a8455497933b56bb8032a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44309 } }
I20260926 13:14:17.852104 2852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:17.852147 2852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:17.852229 2852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:17.852923 2852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93399b8829a8455497933b56bb8032a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44309 } }
I20260926 13:14:17.853039 2852 leader_election.cc:302] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93399b8829a8455497933b56bb8032a6; no voters:
I20260926 13:14:17.853214 2852 leader_election.cc:288] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:17.853313 2857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:17.853497 2857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [term 1 LEADER]: Becoming Leader. State: Replica: 93399b8829a8455497933b56bb8032a6, State: Running, Role: LEADER
I20260926 13:14:17.853601 2852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:17.853641 2857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93399b8829a8455497933b56bb8032a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44309 } }
I20260926 13:14:17.854067 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93399b8829a8455497933b56bb8032a6. Latest consensus state: current_term: 1 leader_uuid: "93399b8829a8455497933b56bb8032a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93399b8829a8455497933b56bb8032a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44309 } } }
I20260926 13:14:17.854249 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:17.854534 2866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 13:14:17 dist-test-slave-f864 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:17.855074 2858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93399b8829a8455497933b56bb8032a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93399b8829a8455497933b56bb8032a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44309 } } }
I20260926 13:14:17.855232 2858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:17.855182 2866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:17.857963 2866 catalog_manager.cc:1347] Generated new cluster ID: a5d8cc7bfec3462698cc5dd0f8d561c3
I20260926 13:14:17.858016 2866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:17.859920 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:17.847745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.0.98.62:44309)
0926 13:14:17.847921 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:17.847930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:17.847966 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:14:17.848410 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:17.848417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:17.848466 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:17.848590 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:17.849507 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:17.850611 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:17.851451 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:17.852060 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:17.852205 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:17.855315 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:17.855319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:17.855320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:17.855332 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:17.857467 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:17.858002 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:17.858006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:17.858006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:17.858153 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:17.858449 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:17.858450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:17.858450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:17.858771 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:17.858946 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:17.859502 (+ 556us) server_negotiation.cc:299] Negotiation successful
0926 13:14:17.859667 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260926 13:14:17.889722 2866 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:17.890367 2866 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:17.894203 2866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93399b8829a8455497933b56bb8032a6: Generated new TSK 0
I20260926 13:14:17.894431 2866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:17.929620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44309
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:18.020345 2878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:18.020594 2878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:18.020637 2878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:18.021808 2878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:18.021879 2878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:18.021927 2878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:18.024062 2878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:18.024351 2878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:18.024595 2878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:18.029819 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:18.029853 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:18.029976 2878 server_base.cc:1034] running on GCE node
W20260926 13:14:18.029821 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:18.030349 2878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:18.030673 2878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:18.031847 2878 hybrid_clock.cc:648] HybridClock initialized: now 1790428458031820 us; error 42 us; skew 500 ppm
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:18.038007 2878 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:18.038686 2878 webserver.cc:460] Webserver started at http://127.0.98.1:38717/ using document root <none> and password file <none>
I20260926 13:14:18.038926 2878 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:18.038996 2878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:18.039139 2878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:18.040395 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "6d87235c6287437b99f04fb6933a11ce"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "5c9c136fa690abfad77050674a15ea3e"
server_key_iv: "c7bad857b8bb02411dc30ebeaade1fb1"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.040814 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6d87235c6287437b99f04fb6933a11ce"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "5c9c136fa690abfad77050674a15ea3e"
server_key_iv: "c7bad857b8bb02411dc30ebeaade1fb1"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.042338 2878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:18.043143 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.043363 2878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:18.043465 2878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "6d87235c6287437b99f04fb6933a11ce"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "5c9c136fa690abfad77050674a15ea3e"
server_key_iv: "c7bad857b8bb02411dc30ebeaade1fb1"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.043557 2878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:18.052083 2878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:18.054361 2878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:18.054531 2878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:18.054790 2878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:18.055138 2878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:18.055187 2878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.055230 2878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:18.055258 2878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.061009 2878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46687
I20260926 13:14:18.061053 3006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46687 every 8 connection(s)
I20260926 13:14:18.061429 2878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:14:18.065919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 2878
I20260926 13:14:18.066023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:18.066231 392 external_mini_cluster.cc:1428] Setting key 76b639458cba81d0fd5a7a4d603fc014
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:18.073279 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:18.062416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50303 (local address 127.0.98.62:44309)
0926 13:14:18.062479 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:18.062481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:18.062714 (+ 233us) server_negotiation.cc:407] Connection header received
0926 13:14:18.063726 (+ 1012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:18.063727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:18.063741 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:18.063796 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:18.064517 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.065051 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.066460 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.066725 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.066785 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:18.069872 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:18.069874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:18.069874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:18.069882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:18.071466 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:18.072125 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:18.072126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:18.072127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:18.072189 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:18.072533 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:18.072533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:18.072534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:18.072648 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:18.072701 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:18.073107 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 13:14:18.073203 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:14:18.073551 3009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:18.062380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44309 (local address 127.0.98.1:50303)
0926 13:14:18.062544 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:18.062583 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:18.063590 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:18.063881 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:18.063885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:18.064182 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:18.064397 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:18.064402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.065166 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.065168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:18.066359 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:18.066362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.066811 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.066812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:18.066945 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:18.067674 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:18.067686 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:18.069752 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:18.071558 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:18.071568 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:18.071571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:18.072016 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:18.072266 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:18.072267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:18.072268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:18.072458 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:18.072715 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:18.072723 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:18.072928 (+ 205us) client_negotiation.cc:769] Sending connection context
0926 13:14:18.073057 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 13:14:18.073261 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 13:14:18.074409 3007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44309
I20260926 13:14:18.074523 3007 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:18.074757 3007 heartbeater.cc:511] Master 127.0.98.62:44309 requested a full tablet report, sending...
I20260926 13:14:18.075353 2816 ts_manager.cc:194] Registered new tserver with Master: 6d87235c6287437b99f04fb6933a11ce (127.0.98.1:46687)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:14:18.075977 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50303
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:18.093842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44309
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:18.176334 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:18.176563 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:18.176585 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:18.177829 3012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:18.177881 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:18.177907 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:18.179548 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:18.179838 3012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:18.180081 3012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:18.185305 3020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:18.185305 3018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:18.185324 3017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:18.185568 3012 server_base.cc:1034] running on GCE node
I20260926 13:14:18.185846 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:18.186048 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:18.187207 3012 hybrid_clock.cc:648] HybridClock initialized: now 1790428458187192 us; error 29 us; skew 500 ppm
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:18.189858 3012 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:18.190330 3012 webserver.cc:460] Webserver started at http://127.0.98.2:43559/ using document root <none> and password file <none>
I20260926 13:14:18.190518 3012 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:18.190558 3012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:18.190657 3012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:18.191689 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "2506676205624b98b33bf40fddd13ac9"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "3d0823659c9b6b2e6c75f6785f2912ae"
server_key_iv: "e00f681a9dc7865902842883ae0928bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.191987 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2506676205624b98b33bf40fddd13ac9"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "3d0823659c9b6b2e6c75f6785f2912ae"
server_key_iv: "e00f681a9dc7865902842883ae0928bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.193262 3012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:14:18.193975 3027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.194150 3012 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:18.194293 3012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "2506676205624b98b33bf40fddd13ac9"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "3d0823659c9b6b2e6c75f6785f2912ae"
server_key_iv: "e00f681a9dc7865902842883ae0928bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.194376 3012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:18.205291 3012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:18.207355 3012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:18.207515 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:18.207782 3012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:18.208096 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:18.208149 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.208181 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:18.208225 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.214087 3012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43949
I20260926 13:14:18.214128 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43949 every 8 connection(s)
I20260926 13:14:18.214473 3012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:18.218964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3012
I20260926 13:14:18.219049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:18.219254 392 external_mini_cluster.cc:1428] Setting key 1722094fb6b14104465fdc5275033884
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:18.224871 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:18.215221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44309 (local address 127.0.98.2:34619)
0926 13:14:18.215407 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:18.215444 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:18.216421 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:18.216698 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:18.216702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:18.216966 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:18.217186 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:18.217191 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.217910 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.217911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:18.218814 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:18.218817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.219318 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.219319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:18.219442 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:18.220023 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:18.220032 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:18.221423 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:18.223167 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:18.223174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:18.223176 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:18.223562 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:18.223762 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:18.223763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:18.223764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:18.223915 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:18.224194 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:18.224199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:18.224381 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 13:14:18.224471 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:14:18.224693 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":92,"threads_started":1}
I20260926 13:14:18.224884 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:18.215252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34619 (local address 127.0.98.62:44309)
0926 13:14:18.215319 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:18.215321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:18.215529 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:14:18.216537 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:18.216539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:18.216551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:18.216610 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:18.217287 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.217808 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.218908 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.219218 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.219307 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:18.221522 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:18.221523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:18.221524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:18.221533 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:18.223076 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:18.223645 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:18.223646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:18.223646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:18.223695 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:18.223988 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:18.223989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:18.223989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:18.224098 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:18.224184 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:18.224525 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 13:14:18.224647 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:14:18.225493 3141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44309
I20260926 13:14:18.225617 3141 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:18.225827 3141 heartbeater.cc:511] Master 127.0.98.62:44309 requested a full tablet report, sending...
I20260926 13:14:18.226316 2816 ts_manager.cc:194] Registered new tserver with Master: 2506676205624b98b33bf40fddd13ac9 (127.0.98.2:43949)
I20260926 13:14:18.226861 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34619
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:18.250432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44309
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:18.332929 3146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:18.333135 3146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:18.333159 3146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:18.334393 3146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:18.334432 3146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:18.334460 3146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:18.336024 3146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:18.336218 3146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:18.336397 3146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:18.341455 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:18.341481 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:18.341665 3146 server_base.cc:1034] running on GCE node
W20260926 13:14:18.341684 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:18.341920 3146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:18.342123 3146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:18.343326 3146 hybrid_clock.cc:648] HybridClock initialized: now 1790428458343289 us; error 52 us; skew 500 ppm
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:18.345891 3146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:18.346414 3146 webserver.cc:460] Webserver started at http://127.0.98.3:42775/ using document root <none> and password file <none>
I20260926 13:14:18.346664 3146 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:18.346731 3146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:18.346897 3146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:18.347988 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "37f6884b3cd147d2938f1fdd89eab32e"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "4ba3e3dd0be914f96ba1e16e6731855f"
server_key_iv: "85cd2b4339586f280019ad8940b2e639"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.348356 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37f6884b3cd147d2938f1fdd89eab32e"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "4ba3e3dd0be914f96ba1e16e6731855f"
server_key_iv: "85cd2b4339586f280019ad8940b2e639"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.349733 3146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:14:18.350605 3161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.350812 3146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:14:18.350921 3146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "37f6884b3cd147d2938f1fdd89eab32e"
format_stamp: "Formatted at 2026-09-26 13:14:18 on dist-test-slave-f864"
server_key: "4ba3e3dd0be914f96ba1e16e6731855f"
server_key_iv: "85cd2b4339586f280019ad8940b2e639"
server_key_version: "encryptionkey@0"
I20260926 13:14:18.350996 3146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:18.365953 3146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:18.368072 3146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:18.368252 3146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:18.368506 3146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:18.368836 3146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:18.368974 3146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.369073 3146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:18.369107 3146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:18.375285 3146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40003
I20260926 13:14:18.375348 3274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40003 every 8 connection(s)
I20260926 13:14:18.375715 3146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:18.385178 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:18.376504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58597 (local address 127.0.98.62:44309)
0926 13:14:18.376586 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:18.376588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:18.376777 (+ 189us) server_negotiation.cc:407] Connection header received
0926 13:14:18.377777 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:18.377779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:18.377791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:18.377855 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:18.378527 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.379039 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.380057 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.380241 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.380293 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:18.382762 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:18.382763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:18.382764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:18.382771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:18.383992 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:18.384375 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:18.384376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:18.384377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:18.384424 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:18.384637 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:18.384637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:18.384638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:18.384739 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:18.384788 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:18.385049 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 13:14:18.385111 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:14:18.385412 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:18.376464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44309 (local address 127.0.98.3:58597)
0926 13:14:18.376645 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:18.376682 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:18.377630 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:18.377921 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:18.377925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:18.378223 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:18.378440 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:18.378444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.379127 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.379128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:18.379970 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:18.379973 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:18.380319 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:18.380320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:18.380431 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:18.381061 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:18.381071 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:18.382663 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:18.384066 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:18.384072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:18.384074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:18.384299 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:18.384481 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:18.384481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:18.384482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:18.384582 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:18.384797 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:18.384802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:18.384932 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 13:14:18.385021 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:14:18.385144 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260926 13:14:18.385739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3146
I20260926 13:14:18.385818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:18.385993 392 external_mini_cluster.cc:1428] Setting key 6189c9f721c33ed3418bcb444d1baf75
I20260926 13:14:18.387379 3275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44309
I20260926 13:14:18.387578 3275 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:18.387884 3275 heartbeater.cc:511] Master 127.0.98.62:44309 requested a full tablet report, sending...
I20260926 13:14:18.388351 2816 ts_manager.cc:194] Registered new tserver with Master: 37f6884b3cd147d2938f1fdd89eab32e (127.0.98.3:40003)
I20260926 13:14:18.389254 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58597
I20260926 13:14:18.398375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:14:18.413525 2855 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:18.412459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.0.98.62:44309)
0926 13:14:18.412517 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:18.412519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:18.412594 (+ 75us) server_negotiation.cc:407] Connection header received
0926 13:14:18.412660 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:18.412661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:18.412669 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:18.412754 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:18.413439 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41354: BlockingRecv error: recv got EOF from 127.0.0.1:41354 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:14:18.414501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 2878
I20260926 13:14:18.419417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3012
I20260926 13:14:18.423781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3146
I20260926 13:14:18.427971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 2784
2026-09-26T13:14:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1892 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[3287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[3287](info): set up 2 sockets
Sep 26 13:14:18 dist-test-slave-f864 krb5kdc[3287](info): commencing operation
krb5kdc: starting...
W20260926 13:14:19.466043 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:19 dist-test-slave-f864 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:19.592911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:19.673803 3300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:19.674029 3300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:19.674053 3300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:19.675307 3300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:19.675364 3300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:19.675379 3300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:19.675392 3300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:19.675405 3300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:19.676901 3300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:19.677114 3300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:19.677291 3300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:19.682739 3308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:19.682819 3305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:19.682744 3306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:19.683137 3300 server_base.cc:1034] running on GCE node
I20260926 13:14:19.683460 3300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:19.683802 3300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:19.684988 3300 hybrid_clock.cc:648] HybridClock initialized: now 1790428459684967 us; error 40 us; skew 500 ppm
Sep 26 13:14:19 dist-test-slave-f864 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:19.687755 3300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:19.688259 3300 webserver.cc:460] Webserver started at http://127.0.98.62:42403/ using document root <none> and password file <none>
I20260926 13:14:19.688517 3300 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:19.688589 3300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:19.688740 3300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:19.689819 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "67c462a2d3ef4ca3baa414aac443c015"
format_stamp: "Formatted at 2026-09-26 13:14:19 on dist-test-slave-f864"
server_key: "525cd5202d05f4cdadb735b8fe8a6e83"
server_key_iv: "1d8024aa48a115603100a06d394911a8"
server_key_version: "encryptionkey@0"
I20260926 13:14:19.690254 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "67c462a2d3ef4ca3baa414aac443c015"
format_stamp: "Formatted at 2026-09-26 13:14:19 on dist-test-slave-f864"
server_key: "525cd5202d05f4cdadb735b8fe8a6e83"
server_key_iv: "1d8024aa48a115603100a06d394911a8"
server_key_version: "encryptionkey@0"
I20260926 13:14:19.691632 3300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:19.692430 3315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:19.692629 3300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:19.692723 3300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "67c462a2d3ef4ca3baa414aac443c015"
format_stamp: "Formatted at 2026-09-26 13:14:19 on dist-test-slave-f864"
server_key: "525cd5202d05f4cdadb735b8fe8a6e83"
server_key_iv: "1d8024aa48a115603100a06d394911a8"
server_key_version: "encryptionkey@0"
I20260926 13:14:19.692806 3300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:19.707857 3300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:19.709834 3300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:19.710011 3300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:19.713729 3300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43845
I20260926 13:14:19.713848 3367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43845 every 8 connection(s)
I20260926 13:14:19.714318 3300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:19.714999 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:19.717185 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015: Bootstrap starting.
I20260926 13:14:19.717777 3368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:19.718004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3300
I20260926 13:14:19.718103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:19.718133 3368 log.cc:824] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:19.718317 392 external_mini_cluster.cc:1428] Setting key 7876ff0a072fdee7879d1f92d4a044a9
I20260926 13:14:19.719040 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015: No bootstrap required, opened a new log
I20260926 13:14:19.721856 3368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c462a2d3ef4ca3baa414aac443c015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260926 13:14:19.722054 3368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:19.722092 3368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c462a2d3ef4ca3baa414aac443c015, State: Initialized, Role: FOLLOWER
I20260926 13:14:19.722285 3368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c462a2d3ef4ca3baa414aac443c015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260926 13:14:19.722379 3368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:19.722411 3368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:19.722452 3368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:19.723165 3368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c462a2d3ef4ca3baa414aac443c015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260926 13:14:19.723261 3368 leader_election.cc:302] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67c462a2d3ef4ca3baa414aac443c015; no voters:
I20260926 13:14:19.723476 3368 leader_election.cc:288] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:19.723505 3373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:19.723675 3373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [term 1 LEADER]: Becoming Leader. State: Replica: 67c462a2d3ef4ca3baa414aac443c015, State: Running, Role: LEADER
I20260926 13:14:19.723769 3368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:19.724117 3373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c462a2d3ef4ca3baa414aac443c015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } }
I20260926 13:14:19.724494 3374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67c462a2d3ef4ca3baa414aac443c015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c462a2d3ef4ca3baa414aac443c015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } } }
I20260926 13:14:19.724592 3374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:19.724717 3374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c462a2d3ef4ca3baa414aac443c015. Latest consensus state: current_term: 1 leader_uuid: "67c462a2d3ef4ca3baa414aac443c015" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c462a2d3ef4ca3baa414aac443c015" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43845 } } }
I20260926 13:14:19.724768 3374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:19.724905 3382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:19.725459 3382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 13:14:19 dist-test-slave-f864 krb5kdc[3287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:19.727451 3382 catalog_manager.cc:1347] Generated new cluster ID: 1b246b800f73415ca05e428ed53b9ffb
I20260926 13:14:19.727512 3382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:19.730726 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:19.718847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.0.98.62:43845)
0926 13:14:19.719050 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:19.719059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:19.719118 (+ 59us) server_negotiation.cc:407] Connection header received
0926 13:14:19.719583 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:19.719590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:19.719639 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:19.719767 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:19.720654 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:19.721665 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:19.722805 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:19.723318 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:19.723461 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:19.726790 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:19.726794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:19.726795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:19.726808 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:19.728952 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:19.729398 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:19.729402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:19.729403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:19.729500 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:19.729737 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:19.729738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:19.729738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:19.730011 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:19.730253 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:19.730339 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:14:19.730499 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":119,"threads_started":1}
I20260926 13:14:19.731779 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:14:19.739990 3382 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:19.740739 3382 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:19.747298 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:19.743483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.0.98.62:43845)
0926 13:14:19.743530 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:19.743532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:19.743542 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:19.743607 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:19.743608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:19.743619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:19.743682 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:19.744097 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:19.744642 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:19.745508 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:19.745764 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:19.745846 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:19.746727 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:19.746729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:19.746729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:19.746964 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:19.746982 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:19.746989 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:19.747065 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:19.747170 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 13:14:19.747203 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:14:19.751767 3382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67c462a2d3ef4ca3baa414aac443c015: Generated new TSK 0
I20260926 13:14:19.751998 3382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:14:19.758632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3300
2026-09-26T13:14:19Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1328 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:19 dist-test-slave-f864 krb5kdc[3401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:19 dist-test-slave-f864 krb5kdc[3401](info): set up 2 sockets
Sep 26 13:14:19 dist-test-slave-f864 krb5kdc[3401](info): commencing operation
krb5kdc: starting...
W20260926 13:14:20.806653 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:20 dist-test-slave-f864 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:20.933557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46263
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:21.020987 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:21.021252 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:21.021289 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:21.022550 3414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:21.022622 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:21.022646 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:21.022667 3414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:21.022693 3414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:21.024257 3414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:21.024566 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:21.024811 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:21.029809 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.029804 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.029804 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:21.029889 3414 server_base.cc:1034] running on GCE node
I20260926 13:14:21.030283 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:21.030616 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:21.031749 3414 hybrid_clock.cc:648] HybridClock initialized: now 1790428461031737 us; error 33 us; skew 500 ppm
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:21.043005 3414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:21.043519 3414 webserver.cc:460] Webserver started at http://127.0.98.62:38509/ using document root <none> and password file <none>
I20260926 13:14:21.043776 3414 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:21.043824 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:21.044003 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:21.045087 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "202e68e9b5414913bf92a9d9280f0f8a"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "019e8197d86f38696e2b84d46a731d27"
server_key_iv: "c2769930ec4a506ec56a41e26c3e3653"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.045455 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "202e68e9b5414913bf92a9d9280f0f8a"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "019e8197d86f38696e2b84d46a731d27"
server_key_iv: "c2769930ec4a506ec56a41e26c3e3653"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.046941 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:21.047717 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.047897 3414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:21.048007 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "202e68e9b5414913bf92a9d9280f0f8a"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "019e8197d86f38696e2b84d46a731d27"
server_key_iv: "c2769930ec4a506ec56a41e26c3e3653"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.048084 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:21.076370 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:21.078454 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:21.078625 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:21.082723 3414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46263
I20260926 13:14:21.082821 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46263 every 8 connection(s)
I20260926 13:14:21.083185 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:21.084188 3482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:21.086572 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a: Bootstrap starting.
I20260926 13:14:21.087159 3482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:21.087450 3482 log.cc:824] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:21.088218 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a: No bootstrap required, opened a new log
I20260926 13:14:21.089207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3414
I20260926 13:14:21.089332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:21.089507 392 external_mini_cluster.cc:1428] Setting key 2bb4abbdf24512434401aefe4059370d
I20260926 13:14:21.090323 3482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202e68e9b5414913bf92a9d9280f0f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46263 } }
I20260926 13:14:21.090510 3482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:21.090543 3482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202e68e9b5414913bf92a9d9280f0f8a, State: Initialized, Role: FOLLOWER
I20260926 13:14:21.090667 3482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202e68e9b5414913bf92a9d9280f0f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46263 } }
I20260926 13:14:21.090745 3482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:21.090777 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:21.090828 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:21.091873 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202e68e9b5414913bf92a9d9280f0f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46263 } }
I20260926 13:14:21.091986 3482 leader_election.cc:302] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 202e68e9b5414913bf92a9d9280f0f8a; no voters:
I20260926 13:14:21.092137 3482 leader_election.cc:288] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:21.092267 3487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:21.092387 3482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:21.092471 3487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [term 1 LEADER]: Becoming Leader. State: Replica: 202e68e9b5414913bf92a9d9280f0f8a, State: Running, Role: LEADER
I20260926 13:14:21.092567 3487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202e68e9b5414913bf92a9d9280f0f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46263 } }
I20260926 13:14:21.092970 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "202e68e9b5414913bf92a9d9280f0f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202e68e9b5414913bf92a9d9280f0f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46263 } } }
I20260926 13:14:21.093057 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:21.093195 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 202e68e9b5414913bf92a9d9280f0f8a. Latest consensus state: current_term: 1 leader_uuid: "202e68e9b5414913bf92a9d9280f0f8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "202e68e9b5414913bf92a9d9280f0f8a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46263 } } }
I20260926 13:14:21.093259 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a [sys.catalog]: This master's current role is: LEADER
W20260926 13:14:21.094362 3503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:21.094417 3503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:21.094486 3496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:21.094713 3496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:21.096766 3496 catalog_manager.cc:1347] Generated new cluster ID: c08474587c394d7ea99d3b844a7c4144
I20260926 13:14:21.096839 3496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:21.102065 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.090109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.98.62:46263)
0926 13:14:21.090317 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:21.090327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:21.090362 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:14:21.090475 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:21.090483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:21.090543 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:21.090691 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:21.091433 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.092874 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.094299 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.094904 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.095034 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.097808 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:21.097812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:21.097813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:21.097837 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:21.099973 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.100369 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.100372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.100372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.100501 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.100704 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.100705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.100705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.101018 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:21.101626 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:21.101702 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:14:21.101868 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260926 13:14:21.118076 3496 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:21.118661 3496 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:21.122596 3496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 202e68e9b5414913bf92a9d9280f0f8a: Generated new TSK 0
I20260926 13:14:21.122787 3496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:21.153810 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46263
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:21.238749 3508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:21.238968 3508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:21.238992 3508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:21.240244 3508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:21.240293 3508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:21.240319 3508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:21.241730 3508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:21.241942 3508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:21.242120 3508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:21.247298 3516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.247269 3514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.247315 3513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:21.247490 3508 server_base.cc:1034] running on GCE node
I20260926 13:14:21.247762 3508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:21.248039 3508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:21.249207 3508 hybrid_clock.cc:648] HybridClock initialized: now 1790428461249207 us; error 42 us; skew 500 ppm
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:21.251943 3508 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:21.252485 3508 webserver.cc:460] Webserver started at http://127.0.98.1:37657/ using document root <none> and password file <none>
I20260926 13:14:21.252727 3508 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:21.252797 3508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:21.252954 3508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:21.254076 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce53a31bf2554f4dbb44c9e7ff384b9c"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "589e812f9a407976f9f542609ce89891"
server_key_iv: "03bba32f792befd3799b32667e32a9b2"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.254508 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce53a31bf2554f4dbb44c9e7ff384b9c"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "589e812f9a407976f9f542609ce89891"
server_key_iv: "03bba32f792befd3799b32667e32a9b2"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.255846 3508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:21.256559 3523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.256726 3508 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:21.256840 3508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "ce53a31bf2554f4dbb44c9e7ff384b9c"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "589e812f9a407976f9f542609ce89891"
server_key_iv: "03bba32f792befd3799b32667e32a9b2"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.256917 3508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:21.272689 3508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:21.274957 3508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:21.275144 3508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:21.275409 3508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:21.275753 3508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:21.275811 3508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.275848 3508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:21.275879 3508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.282066 3508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33193
I20260926 13:14:21.282120 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33193 every 8 connection(s)
I20260926 13:14:21.282483 3508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:21.289405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3508
I20260926 13:14:21.289494 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:21.289692 392 external_mini_cluster.cc:1428] Setting key 72b4ab05b06a535cd3df684ab6c2b2bb
I20260926 13:14:21.292907 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.283322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34493 (local address 127.0.98.62:46263)
0926 13:14:21.283415 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:21.283417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:21.283604 (+ 187us) server_negotiation.cc:407] Connection header received
0926 13:14:21.284126 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:21.284127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:21.284143 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:21.284212 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:21.285232 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.285749 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.286773 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.286981 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.287044 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.289282 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:21.289284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:21.289284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:21.289291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:21.291203 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.291623 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.291624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.291625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.291690 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.292117 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.292118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.292118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.292288 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:21.292342 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:21.292724 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 13:14:21.292844 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:14:21.293129 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.283234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46263 (local address 127.0.98.1:34493)
0926 13:14:21.283436 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:21.283483 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:21.284014 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:21.284329 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:21.284333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:21.284926 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:21.285129 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:21.285133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.285834 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.285835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:21.286681 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:21.286683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.287066 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.287067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:21.287173 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.287676 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:21.287684 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:21.289157 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:21.291299 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:21.291305 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:21.291307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:21.291545 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:21.291868 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:21.291869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:21.291870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:21.292044 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:21.292365 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:21.292371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:21.292555 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 13:14:21.292683 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 13:14:21.292869 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":112,"threads_started":1}
I20260926 13:14:21.293823 3637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46263
I20260926 13:14:21.293947 3637 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:21.294194 3637 heartbeater.cc:511] Master 127.0.98.62:46263 requested a full tablet report, sending...
I20260926 13:14:21.294764 3446 ts_manager.cc:194] Registered new tserver with Master: ce53a31bf2554f4dbb44c9e7ff384b9c (127.0.98.1:33193)
I20260926 13:14:21.295395 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34493
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:21.317428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46263
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:21.399695 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:21.399919 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:21.399943 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:21.401120 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:21.401166 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:21.401194 3642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:21.402649 3642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:21.402926 3642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:21.403174 3642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:21.408238 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.408243 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:21.408349 3642 server_base.cc:1034] running on GCE node
W20260926 13:14:21.408308 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:21.408761 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:21.409029 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:21.410239 3642 hybrid_clock.cc:648] HybridClock initialized: now 1790428461410215 us; error 39 us; skew 500 ppm
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:21.412967 3642 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:21.413492 3642 webserver.cc:460] Webserver started at http://127.0.98.2:40503/ using document root <none> and password file <none>
I20260926 13:14:21.413767 3642 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:21.413851 3642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:21.414012 3642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:21.415184 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "48e6a4043a9942f5af88f1c437d42f9f"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "60d4ca73ea602f8067879ae82adeb841"
server_key_iv: "4632c8faa69f0e6d5dd71658d9040db5"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.415573 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48e6a4043a9942f5af88f1c437d42f9f"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "60d4ca73ea602f8067879ae82adeb841"
server_key_iv: "4632c8faa69f0e6d5dd71658d9040db5"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.417066 3642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:21.417977 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.418251 3642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:21.418355 3642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "48e6a4043a9942f5af88f1c437d42f9f"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "60d4ca73ea602f8067879ae82adeb841"
server_key_iv: "4632c8faa69f0e6d5dd71658d9040db5"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.418452 3642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:21.432957 3642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:21.435340 3642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:21.435650 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:21.435989 3642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:21.436380 3642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:21.436445 3642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.436492 3642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:21.436525 3642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.443209 3642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41373
I20260926 13:14:21.443246 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41373 every 8 connection(s)
I20260926 13:14:21.443620 3642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:21.452350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3642
I20260926 13:14:21.452474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:21.452713 392 external_mini_cluster.cc:1428] Setting key 4afee059c04a05aa4dadb0c200f4926b
I20260926 13:14:21.454202 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.444440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55321 (local address 127.0.98.62:46263)
0926 13:14:21.444510 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:21.444512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:21.444726 (+ 214us) server_negotiation.cc:407] Connection header received
0926 13:14:21.445221 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:21.445222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:21.445234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:21.445303 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:21.446350 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.446858 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.447980 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.448181 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.448240 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.450738 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:21.450740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:21.450740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:21.450748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:21.452370 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.453003 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.453005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.453005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.453077 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.453374 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.453375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.453375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.453538 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:21.453634 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:21.453904 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 13:14:21.454037 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:14:21.454213 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.444395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46263 (local address 127.0.98.2:55321)
0926 13:14:21.444564 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:21.444607 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:21.445103 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:21.445380 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:21.445384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:21.445969 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:21.446259 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:21.446264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.446973 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.446975 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:21.447875 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:21.447878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.448268 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.448270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:21.448405 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.449022 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:21.449031 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:21.450623 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:21.452522 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:21.452529 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:21.452532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:21.452868 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:21.453160 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:21.453161 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:21.453162 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:21.453302 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:21.453633 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:21.453638 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:21.453765 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 13:14:21.453855 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 13:14:21.453991 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260926 13:14:21.454929 3771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46263
I20260926 13:14:21.455025 3771 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:21.455225 3771 heartbeater.cc:511] Master 127.0.98.62:46263 requested a full tablet report, sending...
I20260926 13:14:21.455603 3446 ts_manager.cc:194] Registered new tserver with Master: 48e6a4043a9942f5af88f1c437d42f9f (127.0.98.2:41373)
I20260926 13:14:21.455992 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55321
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:21.482476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46263
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:21.573932 3776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:21.574267 3776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:21.574304 3776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:21.575631 3776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:21.575681 3776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:21.575712 3776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:21.577397 3776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46263
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:21.577606 3776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:21.577805 3776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:21.583362 3782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.583366 3784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:21.583436 3781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:21.583731 3776 server_base.cc:1034] running on GCE node
I20260926 13:14:21.583990 3776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:21.584225 3776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:21.585450 3776 hybrid_clock.cc:648] HybridClock initialized: now 1790428461585446 us; error 47 us; skew 500 ppm
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:21.588719 3776 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:21.589277 3776 webserver.cc:460] Webserver started at http://127.0.98.3:32973/ using document root <none> and password file <none>
I20260926 13:14:21.589535 3776 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:21.589609 3776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:21.589759 3776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:21.590927 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "a20c5c21022d4f6798282030623c2cb4"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "3bc707be37d6ae8aef8cc744ee799873"
server_key_iv: "1961f104e8930acf9a52e06b2a33ccf4"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.591323 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a20c5c21022d4f6798282030623c2cb4"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "3bc707be37d6ae8aef8cc744ee799873"
server_key_iv: "1961f104e8930acf9a52e06b2a33ccf4"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.592998 3776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:14:21.594071 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.594354 3776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:21.594444 3776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "a20c5c21022d4f6798282030623c2cb4"
format_stamp: "Formatted at 2026-09-26 13:14:21 on dist-test-slave-f864"
server_key: "3bc707be37d6ae8aef8cc744ee799873"
server_key_iv: "1961f104e8930acf9a52e06b2a33ccf4"
server_key_version: "encryptionkey@0"
I20260926 13:14:21.594532 3776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:21.620234 3776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:21.623271 3776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:21.623461 3776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:21.623715 3776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:21.624073 3776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:21.624131 3776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.624166 3776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:21.624194 3776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:21.629941 3776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40971
I20260926 13:14:21.630030 3904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40971 every 8 connection(s)
I20260926 13:14:21.630390 3776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:21.638396 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3776
I20260926 13:14:21.638487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:21.638669 392 external_mini_cluster.cc:1428] Setting key 11ed2d941dfc84a0c5a6ed6ec453b259
I20260926 13:14:21.640741 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.631141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38057 (local address 127.0.98.62:46263)
0926 13:14:21.631210 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:21.631212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:21.631428 (+ 216us) server_negotiation.cc:407] Connection header received
0926 13:14:21.632005 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:21.632007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:21.632019 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:21.632072 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:21.633077 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.633590 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.634704 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.634910 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.635001 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.637285 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:21.637287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:21.637287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:21.637297 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:21.639089 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.639686 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.639688 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.639689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.639756 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.640038 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.640039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.640039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.640167 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:21.640248 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:21.640549 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 13:14:21.640646 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:14:21.641080 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.631117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46263 (local address 127.0.98.3:38057)
0926 13:14:21.631301 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:21.631339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:21.631866 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:21.632166 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:21.632169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:21.632769 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:21.632973 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:21.632978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.633699 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.633700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:21.634599 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:21.634602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.635012 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.635014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:21.635136 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.635674 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:21.635682 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:21.637155 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:21.639188 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:21.639196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:21.639199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:21.639584 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:21.639828 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:21.639829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:21.639830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:21.639961 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:21.640239 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:21.640246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:21.640405 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 13:14:21.640528 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 13:14:21.640701 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260926 13:14:21.641814 3905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46263
I20260926 13:14:21.641912 3905 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:21.642141 3905 heartbeater.cc:511] Master 127.0.98.62:46263 requested a full tablet report, sending...
I20260926 13:14:21.642562 3446 ts_manager.cc:194] Registered new tserver with Master: a20c5c21022d4f6798282030623c2cb4 (127.0.98.3:40971)
I20260926 13:14:21.642925 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38057
I20260926 13:14:21.651002 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428461, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:21.675547 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:21.667021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55992 (local address 127.0.98.62:46263)
0926 13:14:21.667121 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:21.667123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:21.667150 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:14:21.667212 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:21.667213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:21.667223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:21.667268 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:21.667908 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.668473 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.669290 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:21.669495 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:21.669563 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:21.672346 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:21.672348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:21.672348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:21.672357 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:21.674069 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.674591 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.674592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.674592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.674652 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:21.674890 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:21.674890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:21.674891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:21.675052 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:21.675131 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:21.675323 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 13:14:21.675401 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 13:14:21.675927 3446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55992
I20260926 13:14:21.676415 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3508
I20260926 13:14:21.680701 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3642
I20260926 13:14:21.685308 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3776
I20260926 13:14:21.689692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3414
2026-09-26T13:14:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1932 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3917](info): set up 2 sockets
Sep 26 13:14:21 dist-test-slave-f864 krb5kdc[3917](info): commencing operation
krb5kdc: starting...
W20260926 13:14:22.738320 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:22 dist-test-slave-f864 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:22.865242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:22.948566 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:22.948823 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:22.948861 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:22.950086 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:22.950156 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:22.950202 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:22.950237 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:22.950263 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:22.951723 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:22.952014 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:22.952248 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:22.957243 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:22.957253 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:22.957305 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:22.957500 3930 server_base.cc:1034] running on GCE node
I20260926 13:14:22.957780 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:22.958083 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:22.959251 3930 hybrid_clock.cc:648] HybridClock initialized: now 1790428462959240 us; error 66 us; skew 500 ppm
Sep 26 13:14:22 dist-test-slave-f864 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:22.967526 3930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:22.968142 3930 webserver.cc:460] Webserver started at http://127.0.98.62:41803/ using document root <none> and password file <none>
I20260926 13:14:22.968364 3930 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:22.968420 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:22.968531 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:22.969717 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "50f120ec0535475c96787eabf6858482"
format_stamp: "Formatted at 2026-09-26 13:14:22 on dist-test-slave-f864"
server_key: "2d0975e7d5042b4d588dfb3f59304525"
server_key_iv: "9055103fffe0e5765b0adc9eb58f375b"
server_key_version: "encryptionkey@0"
I20260926 13:14:22.970077 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "50f120ec0535475c96787eabf6858482"
format_stamp: "Formatted at 2026-09-26 13:14:22 on dist-test-slave-f864"
server_key: "2d0975e7d5042b4d588dfb3f59304525"
server_key_iv: "9055103fffe0e5765b0adc9eb58f375b"
server_key_version: "encryptionkey@0"
I20260926 13:14:22.971756 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:22.972584 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:22.972877 3930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:22.973001 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "50f120ec0535475c96787eabf6858482"
format_stamp: "Formatted at 2026-09-26 13:14:22 on dist-test-slave-f864"
server_key: "2d0975e7d5042b4d588dfb3f59304525"
server_key_iv: "9055103fffe0e5765b0adc9eb58f375b"
server_key_version: "encryptionkey@0"
I20260926 13:14:22.973124 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:22.991133 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:22.994658 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:22.994827 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:23.000291 3930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40741
I20260926 13:14:23.000346 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40741 every 8 connection(s)
I20260926 13:14:23.000841 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:23.002247 3998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:23.005743 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482: Bootstrap starting.
I20260926 13:14:23.006614 3998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:23.006991 3998 log.cc:824] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:23.007875 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482: No bootstrap required, opened a new log
I20260926 13:14:23.010434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 3930
I20260926 13:14:23.010546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:23.010749 392 external_mini_cluster.cc:1428] Setting key 07235fcdff2e016772a7d115731a6f0f
I20260926 13:14:23.010633 3998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f120ec0535475c96787eabf6858482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 13:14:23.010836 3998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:23.010877 3998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50f120ec0535475c96787eabf6858482, State: Initialized, Role: FOLLOWER
I20260926 13:14:23.011049 3998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f120ec0535475c96787eabf6858482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 13:14:23.011143 3998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:23.011183 3998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:23.011229 3998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:23.012428 3998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f120ec0535475c96787eabf6858482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 13:14:23.012552 3998 leader_election.cc:302] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50f120ec0535475c96787eabf6858482; no voters:
I20260926 13:14:23.012708 3998 leader_election.cc:288] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:23.012810 4003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:23.012995 3998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:23.012997 4003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [term 1 LEADER]: Becoming Leader. State: Replica: 50f120ec0535475c96787eabf6858482, State: Running, Role: LEADER
I20260926 13:14:23.013149 4003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f120ec0535475c96787eabf6858482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 13:14:23.013556 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50f120ec0535475c96787eabf6858482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f120ec0535475c96787eabf6858482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } } }
I20260926 13:14:23.013671 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:23.013582 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50f120ec0535475c96787eabf6858482. Latest consensus state: current_term: 1 leader_uuid: "50f120ec0535475c96787eabf6858482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50f120ec0535475c96787eabf6858482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } } }
I20260926 13:14:23.013815 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:23.014513 4012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:23.015168 4012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:23.017891 4012 catalog_manager.cc:1347] Generated new cluster ID: ca23df0c970644b593972b52b282efd4
I20260926 13:14:23.017961 4012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:23.022861 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.011435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.98.62:40741)
0926 13:14:23.011662 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:23.011672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:23.011708 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:14:23.011848 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:23.011858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:23.011928 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:23.012109 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:23.012798 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.014297 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.015399 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.015971 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.016134 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.018985 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:23.018989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:23.018989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:23.019001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:23.021157 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.021521 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.021525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.021525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.021660 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.021860 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.021861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.021861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.022214 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:23.022390 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:23.022461 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:14:23.022612 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260926 13:14:23.025905 4012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:23.026841 4012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:23.030962 4012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50f120ec0535475c96787eabf6858482: Generated new TSK 0
I20260926 13:14:23.031157 4012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:23.063329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:23.146893 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:23.147136 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:23.147177 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:23.148377 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:23.148452 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:23.148500 4024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:23.150008 4024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:23.150307 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:23.150547 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:23.155610 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:23.155630 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:23.155701 4024 server_base.cc:1034] running on GCE node
W20260926 13:14:23.155610 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:23.156221 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:23.156442 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:23.157588 4024 hybrid_clock.cc:648] HybridClock initialized: now 1790428463157568 us; error 43 us; skew 500 ppm
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:23.160348 4024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:23.160856 4024 webserver.cc:460] Webserver started at http://127.0.98.1:33521/ using document root <none> and password file <none>
I20260926 13:14:23.161123 4024 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:23.161198 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:23.161365 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:23.162602 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "45e5874e355f4c858e423fec3a62b3bd"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "a39e926d724afcffbc92d311bad1cd8a"
server_key_iv: "fe7ee924d208a90dbad215aeff442554"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.163010 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "45e5874e355f4c858e423fec3a62b3bd"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "a39e926d724afcffbc92d311bad1cd8a"
server_key_iv: "fe7ee924d208a90dbad215aeff442554"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.164498 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:23.165340 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.165638 4024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:23.165748 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "45e5874e355f4c858e423fec3a62b3bd"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "a39e926d724afcffbc92d311bad1cd8a"
server_key_iv: "fe7ee924d208a90dbad215aeff442554"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.165829 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:23.189795 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:23.192396 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:23.192579 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:23.192844 4024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:23.193182 4024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:23.193236 4024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.193271 4024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:23.193312 4024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.199076 4024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38023
I20260926 13:14:23.199157 4152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38023 every 8 connection(s)
I20260926 13:14:23.199503 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:23.208936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4024
I20260926 13:14:23.209076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:23.209333 392 external_mini_cluster.cc:1428] Setting key 89b4b8475860d6d596b8f93b90fbe7a0
I20260926 13:14:23.210477 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.200307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35359 (local address 127.0.98.62:40741)
0926 13:14:23.200388 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:23.200391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:23.200548 (+ 157us) server_negotiation.cc:407] Connection header received
0926 13:14:23.201563 (+ 1015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:23.201564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:23.201581 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:23.201656 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:23.202432 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.202939 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.204012 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.204248 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.204329 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.206907 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:23.206909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:23.206909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:23.206916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:23.208605 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.209287 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.209288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.209289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.209361 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.209703 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.209704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.209704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.209875 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:23.209936 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:23.210300 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 13:14:23.210410 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 13:14:23.210876 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.200256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.1:35359)
0926 13:14:23.200418 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:23.200455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:23.201401 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:23.201736 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:23.201739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:23.202006 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:23.202315 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:23.202320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.203021 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.203021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:23.203876 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:23.203878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.204387 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.204388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:23.204526 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.205066 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:23.205075 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:23.206764 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:23.208746 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:23.208754 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:23.208757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:23.209156 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:23.209483 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:23.209484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:23.209484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:23.209611 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:23.209937 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:23.209945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:23.210093 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 13:14:23.210246 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 13:14:23.210467 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 13:14:23.211823 4153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260926 13:14:23.211956 4153 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:23.212157 4153 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260926 13:14:23.212739 3962 ts_manager.cc:194] Registered new tserver with Master: 45e5874e355f4c858e423fec3a62b3bd (127.0.98.1:38023)
I20260926 13:14:23.213901 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35359
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:23.238580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:23.321959 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:23.322263 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:23.322309 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:23.323551 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:23.323629 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:23.323678 4158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:23.325244 4158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:23.325537 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:23.325783 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:23.331076 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:23.331135 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:23.331071 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:23.331333 4158 server_base.cc:1034] running on GCE node
I20260926 13:14:23.331559 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:23.331759 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:23.332906 4158 hybrid_clock.cc:648] HybridClock initialized: now 1790428463332878 us; error 41 us; skew 500 ppm
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:23.335614 4158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:23.336079 4158 webserver.cc:460] Webserver started at http://127.0.98.2:37481/ using document root <none> and password file <none>
I20260926 13:14:23.336310 4158 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:23.336359 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:23.336515 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:23.337592 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "ba1eaead95814c1e9fb58603cab1f260"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "804e3f9932a630d10478ca66bb4d9bcf"
server_key_iv: "38dcad7134f9858e3b519ea6e51f56bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.337975 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ba1eaead95814c1e9fb58603cab1f260"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "804e3f9932a630d10478ca66bb4d9bcf"
server_key_iv: "38dcad7134f9858e3b519ea6e51f56bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.339622 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:23.340477 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.340651 4158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:23.340754 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "ba1eaead95814c1e9fb58603cab1f260"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "804e3f9932a630d10478ca66bb4d9bcf"
server_key_iv: "38dcad7134f9858e3b519ea6e51f56bf"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.340845 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:23.353976 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:23.355955 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:23.356127 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:23.356382 4158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:23.356699 4158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:23.356755 4158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.356802 4158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:23.356835 4158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.362799 4158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42675
I20260926 13:14:23.362851 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42675 every 8 connection(s)
I20260926 13:14:23.363224 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:23.363718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4158
I20260926 13:14:23.363833 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:23.364042 392 external_mini_cluster.cc:1428] Setting key aa6415b3188c1afb2e52e04c9167b1e5
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:14:23.373929 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.364028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40323 (local address 127.0.98.62:40741)
0926 13:14:23.364105 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:23.364107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:23.364348 (+ 241us) server_negotiation.cc:407] Connection header received
0926 13:14:23.365418 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:23.365419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:23.365433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:23.365504 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:23.366327 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.366818 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.367810 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.367996 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.368048 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.370808 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:23.370811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:23.370811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:23.370820 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:23.372398 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.372815 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.372817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.372817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.372879 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.373148 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.373149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.373150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.373307 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:23.373391 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:23.373725 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 13:14:23.373814 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:14:23.374042 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.364027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.2:40323)
0926 13:14:23.364191 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:23.364244 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:23.365302 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:23.365562 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:23.365565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:23.365888 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:23.366230 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:23.366238 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.366912 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.366913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:23.367735 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:23.367738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.368072 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.368073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:23.368228 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.368923 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:23.368934 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:23.370708 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:23.372492 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:23.372498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:23.372500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:23.372734 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:23.372977 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:23.372977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:23.372977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:23.373077 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:23.373411 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:23.373417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:23.373564 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 13:14:23.373684 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 13:14:23.373820 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260926 13:14:23.374668 4287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260926 13:14:23.374758 4287 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:23.374904 4287 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260926 13:14:23.375424 3962 ts_manager.cc:194] Registered new tserver with Master: ba1eaead95814c1e9fb58603cab1f260 (127.0.98.2:42675)
I20260926 13:14:23.375841 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:23.393563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:23.474617 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:23.474852 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:23.474879 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:23.476081 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:23.476130 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:23.476158 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:23.477730 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:23.477943 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:23.478138 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:14:23.483916 4292 server_base.cc:1034] running on GCE node
W20260926 13:14:23.483914 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:23.483875 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:23.483880 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:23.484330 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:23.484639 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:23.485812 4292 hybrid_clock.cc:648] HybridClock initialized: now 1790428463485801 us; error 26 us; skew 500 ppm
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:23.488324 4292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:23.488829 4292 webserver.cc:460] Webserver started at http://127.0.98.3:40583/ using document root <none> and password file <none>
I20260926 13:14:23.489054 4292 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:23.489122 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:23.489275 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:23.490401 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "e08c418739504997bc2e362fd073f23e"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "064633cb7389643483c9ea2a6af0e1ec"
server_key_iv: "14250b9e577dd799f3da77551041f8b0"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.490764 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e08c418739504997bc2e362fd073f23e"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "064633cb7389643483c9ea2a6af0e1ec"
server_key_iv: "14250b9e577dd799f3da77551041f8b0"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.492113 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:23.492969 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.493158 4292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:23.493264 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "e08c418739504997bc2e362fd073f23e"
format_stamp: "Formatted at 2026-09-26 13:14:23 on dist-test-slave-f864"
server_key: "064633cb7389643483c9ea2a6af0e1ec"
server_key_iv: "14250b9e577dd799f3da77551041f8b0"
server_key_version: "encryptionkey@0"
I20260926 13:14:23.493337 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:23.505205 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:23.507357 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:23.507527 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:23.507789 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:23.508118 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:23.508172 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.508205 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:23.508255 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:23.514562 4292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34669
I20260926 13:14:23.514657 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34669 every 8 connection(s)
I20260926 13:14:23.514973 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:23.518913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4292
I20260926 13:14:23.519011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:23.519194 392 external_mini_cluster.cc:1428] Setting key 2c6c19e159a34e1ea9e3c00040dacbc6
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:23.524257 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.515742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46883 (local address 127.0.98.62:40741)
0926 13:14:23.515789 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:23.515791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:23.516015 (+ 224us) server_negotiation.cc:407] Connection header received
0926 13:14:23.516939 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:23.516940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:23.516950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:23.517004 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:23.517581 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.518063 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.519441 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.519723 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.519791 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.521862 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:23.521863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:23.521863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:23.521869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:23.523145 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.523495 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.523496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.523496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.523545 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.523750 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.523750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.523750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.523848 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:23.523897 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:23.524137 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 13:14:23.524195 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:14:23.524437 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.515738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.3:46883)
0926 13:14:23.515892 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:23.515927 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:23.516843 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:23.517059 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:23.517068 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:23.517322 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:23.517512 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:23.517516 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.518140 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.518141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:23.519274 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:23.519277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.519806 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.519807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:23.519929 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.520453 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:23.520461 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:23.521781 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:23.523211 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:23.523217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:23.523219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:23.523432 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:23.523600 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:23.523601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:23.523601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:23.523696 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:23.523901 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:23.523905 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:23.524020 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 13:14:23.524109 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 13:14:23.524226 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260926 13:14:23.524904 4421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260926 13:14:23.524991 4421 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:23.525153 4421 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260926 13:14:23.525521 3962 ts_manager.cc:194] Registered new tserver with Master: e08c418739504997bc2e362fd073f23e (127.0.98.3:34669)
I20260926 13:14:23.525897 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46883
I20260926 13:14:23.531452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:23.556836 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.548450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59348 (local address 127.0.98.1:38023)
0926 13:14:23.548627 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:23.548632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:23.548643 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:23.548680 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:23.548683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:23.548697 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:23.548768 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:23.549341 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.550100 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.550898 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.551254 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.551313 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.553567 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:23.553574 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:23.553574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:23.553581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:23.555585 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.555945 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.555950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.555951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.556021 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.556265 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.556266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.556266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.556413 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:23.556541 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:23.556618 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:14:23.556684 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:23.564843 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:23.558238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57758 (local address 127.0.98.62:40741)
0926 13:14:23.558308 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:23.558310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:23.558321 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:23.558371 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:23.558372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:23.558380 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:23.558451 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:23.558877 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.559357 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.560024 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:23.560203 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:23.560255 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:23.562479 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:23.562480 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:23.562480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:23.562487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:23.563790 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.564145 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.564146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.564146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.564195 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:23.564365 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:23.564366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:23.564366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:23.564505 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:23.564577 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:23.564677 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 13:14:23.564737 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 13:14:23.565143 3962 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57758
I20260926 13:14:23.565526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4024
I20260926 13:14:23.569967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4158
I20260926 13:14:23.574275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4292
I20260926 13:14:23.578959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 3930
2026-09-26T13:14:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1889 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[4433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[4433](info): set up 2 sockets
Sep 26 13:14:23 dist-test-slave-f864 krb5kdc[4433](info): commencing operation
krb5kdc: starting...
W20260926 13:14:24.728585 392 mini_kdc.cc:121] Time spent starting KDC: real 1.129s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 13:14:24.834087 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.105s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 13:14:25.021180 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.187s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/test-user.keytab.
W20260926 13:14:25.829663 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.743s user 0.000s sys 0.002s
Sep 26 13:14:25 dist-test-slave-f864 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:25.927047 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:26.009534 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:26.009778 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:26.009806 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:26.010998 4446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:26.011037 4446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:26.011052 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:26.011065 4446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:26.011076 4446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:26.012580 4446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:26.012791 4446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:26.012966 4446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:26.017951 4454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.017951 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.018019 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:26.018126 4446 server_base.cc:1034] running on GCE node
I20260926 13:14:26.018477 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:26.018746 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:26.019887 4446 hybrid_clock.cc:648] HybridClock initialized: now 1790428466019877 us; error 30 us; skew 500 ppm
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:26.022540 4446 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:26.022919 4446 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:26.022969 4446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:26.023061 4446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:26.024096 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "91715bb9ce1b47ac8f05ab0374ff7730"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "e72b829f674f4baec49f6e75a3dee184"
server_key_iv: "15db51063bdcc00a41638a92fce6a757"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.024397 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "91715bb9ce1b47ac8f05ab0374ff7730"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "e72b829f674f4baec49f6e75a3dee184"
server_key_iv: "15db51063bdcc00a41638a92fce6a757"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.025615 4446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:26.026438 4460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.026679 4446 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:26.026789 4446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "91715bb9ce1b47ac8f05ab0374ff7730"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "e72b829f674f4baec49f6e75a3dee184"
server_key_iv: "15db51063bdcc00a41638a92fce6a757"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.026857 4446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:26.054968 4446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:26.057415 4446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:26.057549 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:26.061993 4446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46511
I20260926 13:14:26.062023 4512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46511 every 8 connection(s)
I20260926 13:14:26.062422 4446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:26.062754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4446
I20260926 13:14:26.062867 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:26.063102 392 external_mini_cluster.cc:1428] Setting key cd01a8b54d656184eeb5445f89f4cbae
I20260926 13:14:26.063803 4513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:26.067413 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730: Bootstrap starting.
I20260926 13:14:26.068189 4513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:26.068509 4513 log.cc:824] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:26.069408 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730: No bootstrap required, opened a new log
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:26.072515 4513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260926 13:14:26.072700 4513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:26.072728 4513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91715bb9ce1b47ac8f05ab0374ff7730, State: Initialized, Role: FOLLOWER
I20260926 13:14:26.072906 4513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260926 13:14:26.073012 4513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:26.073064 4513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:26.073119 4513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:26.074059 4513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260926 13:14:26.074225 4513 leader_election.cc:302] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91715bb9ce1b47ac8f05ab0374ff7730; no voters:
I20260926 13:14:26.074486 4513 leader_election.cc:288] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:26.074520 4518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:26.074651 4518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [term 1 LEADER]: Becoming Leader. State: Replica: 91715bb9ce1b47ac8f05ab0374ff7730, State: Running, Role: LEADER
I20260926 13:14:26.074792 4513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:26.074769 4518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } }
I20260926 13:14:26.075150 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } } }
I20260926 13:14:26.075258 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:26.075397 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91715bb9ce1b47ac8f05ab0374ff7730. Latest consensus state: current_term: 1 leader_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91715bb9ce1b47ac8f05ab0374ff7730" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46511 } } }
I20260926 13:14:26.075472 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:26.075742 4523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:26.076316 4523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:26.077313 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.063739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.98.62:46511)
0926 13:14:26.063959 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.063968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.063997 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:14:26.064474 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.064481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.064535 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.064683 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:26.065564 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.066648 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.067828 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.068332 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.068459 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.071836 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:26.071841 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:26.071842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:26.071857 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:26.075189 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.075715 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.075718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.075719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.075833 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.076077 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.076078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.076078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.076457 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:26.076633 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.076877 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.077032 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":110,"threads_started":1}
I20260926 13:14:26.078346 4523 catalog_manager.cc:1347] Generated new cluster ID: e13bb4462b304fee9224ea622f9d7234
I20260926 13:14:26.078418 4523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:26.088059 4523 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:26.088759 4523 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:26.096599 4523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730: Generated new TSK 0
I20260926 13:14:26.096858 4523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:26.133219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:26.261080 4539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:26.261307 4539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:26.261334 4539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:26.262640 4539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:26.262694 4539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:26.262724 4539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:26.264322 4539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:26.264612 4539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:26.264861 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:26.271124 4547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.271081 4545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.271315 4544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:26.271668 4539 server_base.cc:1034] running on GCE node
I20260926 13:14:26.271962 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:26.272218 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:26.273392 4539 hybrid_clock.cc:648] HybridClock initialized: now 1790428466273382 us; error 51 us; skew 500 ppm
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:26.276254 4539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:26.276669 4539 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:26.276748 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:26.276906 4539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:26.278002 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "7bfeb76255294a11a7b57e1c41b8a35a"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "ea77bb454378585e48479cb7c411a2ba"
server_key_iv: "96bb0efe42095294f12fe8c75559b7c1"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.278398 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7bfeb76255294a11a7b57e1c41b8a35a"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "ea77bb454378585e48479cb7c411a2ba"
server_key_iv: "96bb0efe42095294f12fe8c75559b7c1"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.279829 4539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:26.280623 4553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.280800 4539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:26.280907 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "7bfeb76255294a11a7b57e1c41b8a35a"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "ea77bb454378585e48479cb7c411a2ba"
server_key_iv: "96bb0efe42095294f12fe8c75559b7c1"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.280984 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:26.311918 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:26.314023 4539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:26.314229 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:26.314481 4539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:26.314824 4539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:26.314881 4539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.314924 4539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:26.314950 4539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.320732 4539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33137
I20260926 13:14:26.320773 4666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33137 every 8 connection(s)
I20260926 13:14:26.321053 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:26.329902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4539
I20260926 13:14:26.330032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:26.330291 392 external_mini_cluster.cc:1428] Setting key c05d916f69527274626db69dee3b8890
I20260926 13:14:26.330845 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.321767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58771 (local address 127.0.98.62:46511)
0926 13:14:26.321817 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.321818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.322042 (+ 224us) server_negotiation.cc:407] Connection header received
0926 13:14:26.322616 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.322618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.322628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.322694 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:26.323670 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.324193 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.325244 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.325435 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.325489 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.327793 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:26.327794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:26.327795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:26.327801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:26.329175 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.329645 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.329645 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.329646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.329695 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.329981 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.329982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.329982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.330167 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:26.330296 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.330621 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.330761 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:26.330951 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.321729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46511 (local address 127.0.98.1:58771)
0926 13:14:26.321913 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.321956 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.322501 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.322773 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.322776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.323034 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:26.323544 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.323549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.324313 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.324315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.325161 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.325163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.325514 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.325515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.325629 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.326258 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:26.326269 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:26.327705 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:26.329304 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:26.329317 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:26.329319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:26.329529 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:26.329765 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:26.329765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:26.329765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:26.329912 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:26.330288 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:26.330294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:26.330466 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.330585 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.330755 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260926 13:14:26.331422 4667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46511
I20260926 13:14:26.331516 4667 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:26.331686 4667 heartbeater.cc:511] Master 127.0.98.62:46511 requested a full tablet report, sending...
I20260926 13:14:26.332175 4477 ts_manager.cc:194] Registered new tserver with Master: 7bfeb76255294a11a7b57e1c41b8a35a (127.0.98.1:33137)
I20260926 13:14:26.332798 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58771
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:26.358014 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:26.445210 4672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:26.445421 4672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:26.445446 4672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:26.446625 4672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:26.446667 4672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:26.446693 4672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:26.448171 4672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:26.448354 4672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:26.448539 4672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:26.453377 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.453405 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:26.453565 4672 server_base.cc:1034] running on GCE node
W20260926 13:14:26.453387 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:26.453871 4672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:26.454077 4672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:26.455224 4672 hybrid_clock.cc:648] HybridClock initialized: now 1790428466455206 us; error 33 us; skew 500 ppm
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:26.457583 4672 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:26.457862 4672 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:26.457904 4672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:26.457993 4672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:26.459076 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "12bb51f9d0324945873432a4cd788dad"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "1c6fdf8df00ba594c787f6fdcc3cfa50"
server_key_iv: "e3d5af753745b2f2db4d3ace4d28a0c0"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.459662 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12bb51f9d0324945873432a4cd788dad"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "1c6fdf8df00ba594c787f6fdcc3cfa50"
server_key_iv: "e3d5af753745b2f2db4d3ace4d28a0c0"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.461179 4672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:26.461879 4686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.462037 4672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:26.462127 4672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "12bb51f9d0324945873432a4cd788dad"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "1c6fdf8df00ba594c787f6fdcc3cfa50"
server_key_iv: "e3d5af753745b2f2db4d3ace4d28a0c0"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.462231 4672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:26.477488 4672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:26.479674 4672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:26.479802 4672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:26.479991 4672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:26.480386 4672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:26.480425 4672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.480474 4672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:26.480500 4672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.486545 4672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33113
I20260926 13:14:26.486624 4799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33113 every 8 connection(s)
I20260926 13:14:26.486905 4672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:26.495750 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4672
I20260926 13:14:26.495903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:26.496155 392 external_mini_cluster.cc:1428] Setting key 3645f5a7da218fbeedaddcd7e616d07a
I20260926 13:14:26.496618 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.487626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39647 (local address 127.0.98.62:46511)
0926 13:14:26.487698 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.487700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.487904 (+ 204us) server_negotiation.cc:407] Connection header received
0926 13:14:26.488403 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.488404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.488414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.488471 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:26.489371 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.489887 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.491030 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.491233 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.491316 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.493594 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:26.493595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:26.493596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:26.493601 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:26.495060 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.495446 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.495447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.495447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.495497 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.495866 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.495867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.495867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.496027 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:26.496078 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.496449 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.496524 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:14:26.496893 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.487622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46511 (local address 127.0.98.2:39647)
0926 13:14:26.487778 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.487815 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.488313 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.488528 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.488531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.488792 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:26.489287 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.489292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.490000 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.490001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.490931 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.490933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.491341 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.491342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.491445 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.491998 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:26.492008 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:26.493499 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:26.495140 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:26.495146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:26.495148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:26.495375 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:26.495586 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:26.495587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:26.495588 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:26.495748 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:26.496090 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:26.496097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:26.496278 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.496408 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.496618 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260926 13:14:26.497364 4800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46511
I20260926 13:14:26.497483 4800 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:26.497665 4800 heartbeater.cc:511] Master 127.0.98.62:46511 requested a full tablet report, sending...
I20260926 13:14:26.498087 4477 ts_manager.cc:194] Registered new tserver with Master: 12bb51f9d0324945873432a4cd788dad (127.0.98.2:33113)
I20260926 13:14:26.498625 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39647
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:26.524914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:26.605561 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:26.605780 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:26.605805 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:26.607002 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:26.607046 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:26.607074 4805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:26.608518 4805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:26.608717 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:26.608892 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:26.613981 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.613968 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:26.613986 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:26.614203 4805 server_base.cc:1034] running on GCE node
I20260926 13:14:26.614486 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:26.614753 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:26.615929 4805 hybrid_clock.cc:648] HybridClock initialized: now 1790428466615907 us; error 41 us; skew 500 ppm
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:26.618733 4805 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:26.619120 4805 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:26.619201 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:26.619362 4805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:26.620538 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6c55d3020154bfaa58babdb8c76b056"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "f100a0cbe61d5a1d1ca2116378a8f70a"
server_key_iv: "14aacbaabf4bef00b5763df5142081a5"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.620929 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6c55d3020154bfaa58babdb8c76b056"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "f100a0cbe61d5a1d1ca2116378a8f70a"
server_key_iv: "14aacbaabf4bef00b5763df5142081a5"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.622416 4805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:14:26.623265 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.623464 4805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:26.623575 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "f6c55d3020154bfaa58babdb8c76b056"
format_stamp: "Formatted at 2026-09-26 13:14:26 on dist-test-slave-f864"
server_key: "f100a0cbe61d5a1d1ca2116378a8f70a"
server_key_iv: "14aacbaabf4bef00b5763df5142081a5"
server_key_version: "encryptionkey@0"
I20260926 13:14:26.623658 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:26.670405 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:26.672848 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:26.673039 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:26.673319 4805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:26.673662 4805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:26.673720 4805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.673767 4805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:26.673812 4805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:26.679986 4805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34367
I20260926 13:14:26.680069 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34367 every 8 connection(s)
I20260926 13:14:26.680375 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:14:26 dist-test-slave-f864 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:26.689538 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.681175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33031 (local address 127.0.98.62:46511)
0926 13:14:26.681241 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.681243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.681388 (+ 145us) server_negotiation.cc:407] Connection header received
0926 13:14:26.681874 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.681875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.681885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.681949 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:26.683014 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.683564 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.684594 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.684794 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.684850 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.686998 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:26.687000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:26.687000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:26.687006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:26.688409 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.688762 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.688763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.688763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.688811 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.689010 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.689010 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.689011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.689108 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:26.689158 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.689415 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.689473 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:14:26.689757 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.681098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46511 (local address 127.0.98.3:33031)
0926 13:14:26.681254 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.681298 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.681775 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.682027 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.682031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.682382 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:26.682905 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.682910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.683640 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.683641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.684523 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.684525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.684867 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.684868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.684960 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.685435 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:26.685443 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:26.686905 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:26.688483 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:26.688489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:26.688490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:26.688698 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:26.688866 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:26.688866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:26.688867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:26.688957 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:26.689160 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:26.689164 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:26.689311 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.689392 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.689506 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
I20260926 13:14:26.690330 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4805
I20260926 13:14:26.690339 4933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46511
I20260926 13:14:26.690423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:26.690440 4933 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:26.690578 4933 heartbeater.cc:511] Master 127.0.98.62:46511 requested a full tablet report, sending...
I20260926 13:14:26.690637 392 external_mini_cluster.cc:1428] Setting key db2a8ae1cc37703736883b495282dd20
I20260926 13:14:26.691026 4476 ts_manager.cc:194] Registered new tserver with Master: f6c55d3020154bfaa58babdb8c76b056 (127.0.98.3:34367)
I20260926 13:14:26.691425 4476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33031
I20260926 13:14:26.691514 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:14:26.698504 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.692764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46764 (local address 127.0.98.62:46511)
0926 13:14:26.692819 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.692821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.692942 (+ 121us) server_negotiation.cc:407] Connection header received
0926 13:14:26.693016 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.693017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.693026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.693069 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:26.693638 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.694109 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.694883 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.695072 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.695146 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.696270 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:26.696271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:26.696271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:26.696278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:26.697348 (+ 1070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.697692 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.697693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.697693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.697748 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:26.697944 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:26.697945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:26.697945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:26.698091 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:26.698215 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.698325 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.698405 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:14:26.700326 4476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:26.702001 4476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:26.709044 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.704716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33113 (local address 127.0.0.1:58046)
0926 13:14:26.704987 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.704996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.705038 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.705504 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.705507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.705510 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:26.705663 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.705665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.706607 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.706609 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.707822 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.707824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.708710 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.708711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.708799 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.708814 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.708893 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.708957 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":46,"thread_start_us":132,"threads_started":1}
I20260926 13:14:26.709326 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.704716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34367 (local address 127.0.0.1:37572)
0926 13:14:26.705138 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.705145 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.705185 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.705516 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.705518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.705520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:26.705604 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.705607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.707091 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.707092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.708078 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.708080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.709054 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.709066 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.709145 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.709149 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.709219 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.709260 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":2,"spinlock_wait_cycles":10112,"thread_start_us":134,"threads_started":1}
I20260926 13:14:26.709066 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.705018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58046 (local address 127.0.98.2:33113)
0926 13:14:26.705319 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.705324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.705335 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:26.705380 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.705382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.705399 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.705603 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:26.705730 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.706510 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.707908 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.708599 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.708694 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.708757 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.708923 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.708973 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":141,"threads_started":1}
I20260926 13:14:26.709591 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.705002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37572 (local address 127.0.98.3:34367)
0926 13:14:26.705208 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.705214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.705229 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:14:26.705278 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.705280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.705299 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.705400 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:26.705705 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.706960 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.708186 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.708899 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.708966 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.709019 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.709434 (+ 415us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.709487 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260926 13:14:26.710347 4867 tablet_service.cc:1467] Processing CreateTablet for tablet d853099ae9874f86826f49cf96276ced (DEFAULT_TABLE table=test-table [id=1e455852e6014ba8bb538b62974e111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:26.710739 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d853099ae9874f86826f49cf96276ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:26.710737 4734 tablet_service.cc:1467] Processing CreateTablet for tablet d853099ae9874f86826f49cf96276ced (DEFAULT_TABLE table=test-table [id=1e455852e6014ba8bb538b62974e111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:26.711056 4734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d853099ae9874f86826f49cf96276ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:26.714222 4951 tablet_bootstrap.cc:492] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Bootstrap starting.
I20260926 13:14:26.714367 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.704711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33137 (local address 127.0.0.1:40164)
0926 13:14:26.704905 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.704951 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.705023 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.705537 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.705538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.705539 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:26.705629 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.705630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.708150 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.708150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.708907 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.708909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.709766 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.709767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.714179 (+ 4412us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.714187 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.714245 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.714297 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260926 13:14:26.714406 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.704819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.0.98.1:33137)
0926 13:14:26.705297 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.705301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.705311 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:26.705351 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.705353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.705369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.705445 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:26.705719 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.706961 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.708994 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.709569 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.709676 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.709734 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.714278 (+ 4544us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.714328 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":89,"threads_started":1}
I20260926 13:14:26.714994 4601 tablet_service.cc:1467] Processing CreateTablet for tablet d853099ae9874f86826f49cf96276ced (DEFAULT_TABLE table=test-table [id=1e455852e6014ba8bb538b62974e111a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:26.715075 4951 tablet_bootstrap.cc:654] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:26.715296 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d853099ae9874f86826f49cf96276ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:26.715623 4951 log.cc:824] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:26.716004 4950 tablet_bootstrap.cc:492] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Bootstrap starting.
I20260926 13:14:26.716564 4950 tablet_bootstrap.cc:654] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:26.717048 4950 log.cc:824] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:26.717609 4951 tablet_bootstrap.cc:492] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: No bootstrap required, opened a new log
I20260926 13:14:26.717702 4951 ts_tablet_manager.cc:1397] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:14:26.717823 4950 tablet_bootstrap.cc:492] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: No bootstrap required, opened a new log
I20260926 13:14:26.717892 4950 ts_tablet_manager.cc:1397] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:26.719244 4951 raft_consensus.cc:348] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.719401 4951 raft_consensus.cc:374] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:26.719449 4951 raft_consensus.cc:729] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c55d3020154bfaa58babdb8c76b056, State: Initialized, Role: FOLLOWER
I20260926 13:14:26.719343 4950 raft_consensus.cc:348] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.719491 4950 raft_consensus.cc:374] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:26.719534 4950 raft_consensus.cc:729] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bb51f9d0324945873432a4cd788dad, State: Initialized, Role: FOLLOWER
I20260926 13:14:26.719607 4951 consensus_queue.cc:260] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.719661 4950 consensus_queue.cc:260] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.719910 4933 heartbeater.cc:503] Master 127.0.98.62:46511 was elected leader, sending a full tablet report...
I20260926 13:14:26.719923 4800 heartbeater.cc:503] Master 127.0.98.62:46511 was elected leader, sending a full tablet report...
I20260926 13:14:26.719933 4950 ts_tablet_manager.cc:1428] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:26.720050 4954 tablet_bootstrap.cc:492] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: Bootstrap starting.
I20260926 13:14:26.719901 4951 ts_tablet_manager.cc:1428] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:26.720772 4954 tablet_bootstrap.cc:654] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:26.721349 4954 log.cc:824] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:26.722078 4954 tablet_bootstrap.cc:492] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: No bootstrap required, opened a new log
I20260926 13:14:26.722136 4954 ts_tablet_manager.cc:1397] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:26.723497 4954 raft_consensus.cc:348] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.723632 4954 raft_consensus.cc:374] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:26.723719 4954 raft_consensus.cc:729] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bfeb76255294a11a7b57e1c41b8a35a, State: Initialized, Role: FOLLOWER
I20260926 13:14:26.723840 4954 consensus_queue.cc:260] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.724123 4954 ts_tablet_manager.cc:1428] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:14:26.724133 4667 heartbeater.cc:503] Master 127.0.98.62:46511 was elected leader, sending a full tablet report...
W20260926 13:14:26.737437 4801 tablet.cc:2369] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:14:26.821776 4668 tablet.cc:2369] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:14:26.863301 4955 raft_consensus.cc:482] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:26.863436 4955 raft_consensus.cc:504] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.863847 4955 leader_election.cc:288] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12bb51f9d0324945873432a4cd788dad (127.0.98.2:33113), 7bfeb76255294a11a7b57e1c41b8a35a (127.0.98.1:33137)
I20260926 13:14:26.866928 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.864057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33113 (local address 127.0.98.3:57807)
0926 13:14:26.864129 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.864141 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.864191 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.864341 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.864342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.864344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:26.864418 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.864420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.865197 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.865199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.866081 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.866083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.866672 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.866673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.866760 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.866764 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.866824 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.866868 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 13:14:26.866964 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.864156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57807 (local address 127.0.98.2:33113)
0926 13:14:26.864200 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.864202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.864213 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:14:26.864245 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.864245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.864257 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.864337 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:26.864514 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.865105 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.866201 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.866589 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.866644 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.866680 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.866846 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.866881 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:26.867117 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.864165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51345 (local address 127.0.98.1:33137)
0926 13:14:26.864216 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.864218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.864343 (+ 125us) server_negotiation.cc:407] Connection header received
0926 13:14:26.864421 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.864422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.864435 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.864491 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:26.864655 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.865131 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.866247 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.866583 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.866825 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.866836 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.867000 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.867041 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:14:26.867127 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.864098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33137 (local address 127.0.98.3:51345)
0926 13:14:26.864271 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.864280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:26.864358 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:26.864510 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:26.864511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:26.864512 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:26.864581 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.864583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.865263 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.865264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.866102 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:26.866104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.866850 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.866851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:26.866918 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.866921 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:26.866979 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:14:26.867019 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
I20260926 13:14:26.867367 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d853099ae9874f86826f49cf96276ced" candidate_uuid: "f6c55d3020154bfaa58babdb8c76b056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bb51f9d0324945873432a4cd788dad" is_pre_election: true
I20260926 13:14:26.867535 4754 raft_consensus.cc:2454] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c55d3020154bfaa58babdb8c76b056 in term 0.
I20260926 13:14:26.867523 4621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d853099ae9874f86826f49cf96276ced" candidate_uuid: "f6c55d3020154bfaa58babdb8c76b056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" is_pre_election: true
I20260926 13:14:26.867713 4621 raft_consensus.cc:2454] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c55d3020154bfaa58babdb8c76b056 in term 0.
I20260926 13:14:26.867805 4823 leader_election.cc:302] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12bb51f9d0324945873432a4cd788dad, f6c55d3020154bfaa58babdb8c76b056; no voters:
I20260926 13:14:26.867980 4955 raft_consensus.cc:2781] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:26.868031 4955 raft_consensus.cc:482] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:26.868075 4955 raft_consensus.cc:3037] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:26.868768 4955 raft_consensus.cc:504] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.868914 4955 leader_election.cc:288] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [CANDIDATE]: Term 1 election: Requested vote from peers 12bb51f9d0324945873432a4cd788dad (127.0.98.2:33113), 7bfeb76255294a11a7b57e1c41b8a35a (127.0.98.1:33137)
I20260926 13:14:26.869103 4621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d853099ae9874f86826f49cf96276ced" candidate_uuid: "f6c55d3020154bfaa58babdb8c76b056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bfeb76255294a11a7b57e1c41b8a35a"
I20260926 13:14:26.869061 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d853099ae9874f86826f49cf96276ced" candidate_uuid: "f6c55d3020154bfaa58babdb8c76b056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bb51f9d0324945873432a4cd788dad"
I20260926 13:14:26.869169 4621 raft_consensus.cc:3037] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:26.869215 4754 raft_consensus.cc:3037] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:26.869872 4621 raft_consensus.cc:2454] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c55d3020154bfaa58babdb8c76b056 in term 1.
I20260926 13:14:26.870118 4754 raft_consensus.cc:2454] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c55d3020154bfaa58babdb8c76b056 in term 1.
I20260926 13:14:26.870071 4820 leader_election.cc:302] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7bfeb76255294a11a7b57e1c41b8a35a, f6c55d3020154bfaa58babdb8c76b056; no voters:
I20260926 13:14:26.870249 4955 raft_consensus.cc:2781] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:26.870419 4955 raft_consensus.cc:686] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 1 LEADER]: Becoming Leader. State: Replica: f6c55d3020154bfaa58babdb8c76b056, State: Running, Role: LEADER
I20260926 13:14:26.870528 4955 consensus_queue.cc:237] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } }
I20260926 13:14:26.871392 4477 catalog_manager.cc:5495] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c55d3020154bfaa58babdb8c76b056 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f6c55d3020154bfaa58babdb8c76b056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c55d3020154bfaa58babdb8c76b056" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34367 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:26.891194 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.888549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37582 (local address 127.0.98.3:34367)
0926 13:14:26.888620 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:26.888621 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:26.888630 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:26.888665 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:26.888666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:26.888686 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:26.888757 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:26.888933 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.889463 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.890288 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:26.890481 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:26.890542 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:26.890779 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:26.890887 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:26.890965 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:26.891084 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 13:14:26.891105 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:14:26.893342 4621 raft_consensus.cc:1261] T d853099ae9874f86826f49cf96276ced P 7bfeb76255294a11a7b57e1c41b8a35a [term 1 FOLLOWER]: Refusing update from remote peer f6c55d3020154bfaa58babdb8c76b056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:26.893406 4754 raft_consensus.cc:1261] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 1 FOLLOWER]: Refusing update from remote peer f6c55d3020154bfaa58babdb8c76b056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:26.893795 4955 consensus_queue.cc:1035] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bfeb76255294a11a7b57e1c41b8a35a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:26.893990 4960 consensus_queue.cc:1035] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bb51f9d0324945873432a4cd788dad" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:26.896490 4963 mvcc.cc:205] Tried to move back new op lower bound from 7333595000392003584 to 7333595000302342144. Current Snapshot: MvccSnapshot[applied={T|T < 7333595000392003584}]
I20260926 13:14:26.896533 4965 mvcc.cc:205] Tried to move back new op lower bound from 7333595000392003584 to 7333595000302342144. Current Snapshot: MvccSnapshot[applied={T|T < 7333595000392003584}]
I20260926 13:14:26.899410 4476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:26.899510 4476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:26.900321 4476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91715bb9ce1b47ac8f05ab0374ff7730: Sending DeleteTablet for 3 replicas of tablet d853099ae9874f86826f49cf96276ced
I20260926 13:14:26.900641 4867 tablet_service.cc:1514] Processing DeleteTablet for tablet d853099ae9874f86826f49cf96276ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37572
I20260926 13:14:26.900692 4734 tablet_service.cc:1514] Processing DeleteTablet for tablet d853099ae9874f86826f49cf96276ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58046
I20260926 13:14:26.900691 4601 tablet_service.cc:1514] Processing DeleteTablet for tablet d853099ae9874f86826f49cf96276ced with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40164
I20260926 13:14:26.901005 4974 tablet_replica.cc:330] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: stopping tablet replica
I20260926 13:14:26.901160 4974 raft_consensus.cc:2229] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:26.901212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4539
I20260926 13:14:26.902233 4974 raft_consensus.cc:2258] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:26.902494 4973 tablet_replica.cc:330] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: stopping tablet replica
I20260926 13:14:26.902593 4973 raft_consensus.cc:2229] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:26.902733 4973 raft_consensus.cc:2258] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:26.903136 4973 ts_tablet_manager.cc:1905] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:26.904819 4973 ts_tablet_manager.cc:1918] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:26.904909 4973 log.cc:1196] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/d853099ae9874f86826f49cf96276ced
I20260926 13:14:26.905165 4973 ts_tablet_manager.cc:1939] T d853099ae9874f86826f49cf96276ced P f6c55d3020154bfaa58babdb8c76b056: Deleting consensus metadata
I20260926 13:14:26.905550 4463 catalog_manager.cc:4841] TS f6c55d3020154bfaa58babdb8c76b056 (127.0.98.3:34367): tablet d853099ae9874f86826f49cf96276ced (table test-table [id=1e455852e6014ba8bb538b62974e111a]) successfully deleted
W20260926 13:14:26.905558 4461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33137 (error 108)
I20260926 13:14:26.906001 4974 ts_tablet_manager.cc:1905] T d853099ae9874f86826f49cf96276ced P 12bb51f9d0324945873432a4cd788dad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:26.906116 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.906006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33137 (local address 127.0.0.1:40172)
0926 13:14:26.906046 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.906085 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 13:14:26.906368 4461 catalog_manager.cc:4584] TS 7bfeb76255294a11a7b57e1c41b8a35a (127.0.98.1:33137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d853099ae9874f86826f49cf96276ced: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33137: connect: Connection refused (error 111)
I20260926 13:14:26.906864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4672
I20260926 13:14:26.914127 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:26.913959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33113 (local address 127.0.0.1:58048)
0926 13:14:26.914024 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:26.914055 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:14:26.914270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4805
I20260926 13:14:26.918318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4446
2026-09-26T13:14:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3340 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T13:14:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:26Z Disabled control of system clock
I20260926 13:14:26.946650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41291
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:14:27.031059 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:27.031279 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:27.031301 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:27.032431 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:27.032474 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:27.032488 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:27.032501 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:27.032513 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:27.033866 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41291
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:27.034070 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:27.034404 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:27.039304 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.039316 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.039323 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.039459 4980 server_base.cc:1034] running on GCE node
I20260926 13:14:27.039932 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:27.040295 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:27.041469 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790428467041465 us; error 41 us; skew 500 ppm
I20260926 13:14:27.042809 4980 webserver.cc:460] Webserver started at http://127.0.98.62:34353/ using document root <none> and password file <none>
I20260926 13:14:27.043053 4980 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:27.043126 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:27.043339 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:27.044474 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "1d3d1e3f810449a3bfa5a53f57893d1e"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "5eea55054f95a8ea9b548a0b3d71e700"
server_key_iv: "24aebe622bdd7b6c744e6a1c0035f38e"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.044862 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d3d1e3f810449a3bfa5a53f57893d1e"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "5eea55054f95a8ea9b548a0b3d71e700"
server_key_iv: "24aebe622bdd7b6c744e6a1c0035f38e"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.046345 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 13:14:27.047212 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.047416 4980 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:27.047518 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "1d3d1e3f810449a3bfa5a53f57893d1e"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "5eea55054f95a8ea9b548a0b3d71e700"
server_key_iv: "24aebe622bdd7b6c744e6a1c0035f38e"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.047611 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:27.051084 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:27.051373 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:27.051522 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:27.055371 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41291
I20260926 13:14:27.055440 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41291 every 8 connection(s)
I20260926 13:14:27.055851 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:27.056764 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:27.059584 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e: Bootstrap starting.
I20260926 13:14:27.060132 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:27.060372 5047 log.cc:824] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:27.061062 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e: No bootstrap required, opened a new log
I20260926 13:14:27.061335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 4980
I20260926 13:14:27.061426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:27.061618 392 external_mini_cluster.cc:1428] Setting key 74c07f2f65bf82c0b17ea021175bcd2a
I20260926 13:14:27.063238 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:14:27.063375 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:27.063400 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3d1e3f810449a3bfa5a53f57893d1e, State: Initialized, Role: FOLLOWER
I20260926 13:14:27.063527 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:14:27.063611 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:27.063656 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:27.063758 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:27.064520 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:14:27.064666 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d3d1e3f810449a3bfa5a53f57893d1e; no voters:
I20260926 13:14:27.064860 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:27.064898 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:27.065009 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [term 1 LEADER]: Becoming Leader. State: Replica: 1d3d1e3f810449a3bfa5a53f57893d1e, State: Running, Role: LEADER
I20260926 13:14:27.065112 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:27.065151 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } }
I20260926 13:14:27.065240 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.062230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.0.98.62:41291)
0926 13:14:27.062431 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.062439 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.062464 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:14:27.063250 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.063259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.063335 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.063576 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.064325 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.064329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.064329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.064333 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.064374 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.064386 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.064447 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.064542 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.064642 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260926 13:14:27.065589 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3d1e3f810449a3bfa5a53f57893d1e. Latest consensus state: current_term: 1 leader_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } } }
I20260926 13:14:27.065707 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:27.065605 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d3d1e3f810449a3bfa5a53f57893d1e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41291 } } }
I20260926 13:14:27.065865 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:27.065907 5061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:27.066572 5061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:27.068158 5061 catalog_manager.cc:1347] Generated new cluster ID: 55615c929c93410e9d8e5180a05bdc22
I20260926 13:14:27.068222 5061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:27.086195 5061 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:27.086418 5061 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:27.091742 5061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e: Generated new TSK 0
I20260926 13:14:27.091997 5061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:14:27.098791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41291
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:14:27.207619 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:27.207836 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:27.207859 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:27.209075 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:27.209120 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:27.209149 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:27.210541 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:27.210804 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:27.211108 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:27.216308 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.216331 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.216501 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.216575 5071 server_base.cc:1034] running on GCE node
I20260926 13:14:27.216775 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:27.217031 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:27.218221 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790428467218202 us; error 36 us; skew 500 ppm
I20260926 13:14:27.219547 5071 webserver.cc:460] Webserver started at http://127.0.98.1:34295/ using document root <none> and password file <none>
I20260926 13:14:27.219815 5071 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:27.219887 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:27.220047 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:27.221113 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "60027dbfdaa14a37a8d865f57cf8da08"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "a5af4a424472b64ee59c6ef8e1610e13"
server_key_iv: "1b29366f6718d55c9a34cf11839e4137"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.221488 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "60027dbfdaa14a37a8d865f57cf8da08"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "a5af4a424472b64ee59c6ef8e1610e13"
server_key_iv: "1b29366f6718d55c9a34cf11839e4137"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.222959 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:27.223802 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.223980 5071 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:27.224085 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "60027dbfdaa14a37a8d865f57cf8da08"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "a5af4a424472b64ee59c6ef8e1610e13"
server_key_iv: "1b29366f6718d55c9a34cf11839e4137"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.224211 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:27.227672 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:27.227952 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:27.228097 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:27.228307 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:27.228619 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:27.228672 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.228715 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:27.228742 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.234577 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36407
I20260926 13:14:27.234617 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36407 every 8 connection(s)
I20260926 13:14:27.234956 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:14:27.238152 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.235685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50643 (local address 127.0.98.62:41291)
0926 13:14:27.235810 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.235813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.235976 (+ 163us) server_negotiation.cc:407] Connection header received
0926 13:14:27.236524 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.236525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.236533 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.236583 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.237861 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.237862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.237863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.237867 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.237885 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.237895 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.237957 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.238065 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.238083 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 13:14:27.238376 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.235653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41291 (local address 127.0.98.1:50643)
0926 13:14:27.235850 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.235885 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.236402 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.236675 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.236679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.236687 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.237698 (+ 1011us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.237698 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.237728 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.237756 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.237968 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.237974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.237979 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.238063 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.238128 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260926 13:14:27.238955 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41291
I20260926 13:14:27.239114 5199 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:27.239315 5199 heartbeater.cc:511] Master 127.0.98.62:41291 requested a full tablet report, sending...
I20260926 13:14:27.239862 5011 ts_manager.cc:194] Registered new tserver with Master: 60027dbfdaa14a37a8d865f57cf8da08 (127.0.98.1:36407)
I20260926 13:14:27.244540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5071
I20260926 13:14:27.244697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:27.244931 392 external_mini_cluster.cc:1428] Setting key 8f8560686e589c64cfb644d2cb4b2439
I20260926 13:14:27.246112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41291
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:14:27.327576 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:27.327832 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:27.327870 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:27.329053 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:27.329097 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:27.329129 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:27.330518 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:27.330739 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:27.330997 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:27.336629 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.336575 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.336771 5202 server_base.cc:1034] running on GCE node
W20260926 13:14:27.336596 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.337097 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:27.337388 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:27.338541 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790428467338527 us; error 29 us; skew 500 ppm
I20260926 13:14:27.339877 5202 webserver.cc:460] Webserver started at http://127.0.98.2:33049/ using document root <none> and password file <none>
I20260926 13:14:27.340152 5202 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:27.340225 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:27.340381 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:27.341456 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "a57dc8b4c43744edbbf935381c77bb3b"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "6f8360322f03fd7b5e6a034a5341de8f"
server_key_iv: "53cb256617b76c67bc7af00bf822c79e"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.341833 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a57dc8b4c43744edbbf935381c77bb3b"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "6f8360322f03fd7b5e6a034a5341de8f"
server_key_iv: "53cb256617b76c67bc7af00bf822c79e"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.343245 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:14:27.343974 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.344147 5202 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:27.344246 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "a57dc8b4c43744edbbf935381c77bb3b"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "6f8360322f03fd7b5e6a034a5341de8f"
server_key_iv: "53cb256617b76c67bc7af00bf822c79e"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.344326 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:27.347736 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:27.347998 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:27.348143 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:27.348344 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:27.348639 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:27.348692 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.348730 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:27.348759 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.354694 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33517
I20260926 13:14:27.354734 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33517 every 8 connection(s)
I20260926 13:14:27.355162 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:27.358623 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.355931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51815 (local address 127.0.98.62:41291)
0926 13:14:27.356001 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.356003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.356234 (+ 231us) server_negotiation.cc:407] Connection header received
0926 13:14:27.356722 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.356724 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.356729 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.356805 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.358042 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.358043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.358044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.358048 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.358064 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.358069 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.358211 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.358531 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.358558 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:14:27.358918 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.355918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41291 (local address 127.0.98.2:51815)
0926 13:14:27.356097 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.356139 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.356611 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.356861 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.356864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.356872 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.357878 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.357878 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.357909 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.357937 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.358386 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.358393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.358397 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.358503 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.358579 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260926 13:14:27.359448 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41291
I20260926 13:14:27.359560 5330 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:27.359700 5330 heartbeater.cc:511] Master 127.0.98.62:41291 requested a full tablet report, sending...
I20260926 13:14:27.360147 5011 ts_manager.cc:194] Registered new tserver with Master: a57dc8b4c43744edbbf935381c77bb3b (127.0.98.2:33517)
I20260926 13:14:27.360968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5202
I20260926 13:14:27.361057 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:27.361213 392 external_mini_cluster.cc:1428] Setting key 45a94a180529d75174402960796bf4a5
I20260926 13:14:27.362331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41291
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 13:14:27.452776 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:27.453037 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:27.453078 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:27.454454 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:27.454519 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:27.454566 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:27.456183 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:27.456472 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:27.456789 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:27.461922 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.462083 5333 server_base.cc:1034] running on GCE node
W20260926 13:14:27.461967 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.461972 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.462364 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:27.462649 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:27.463850 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790428467463838 us; error 47 us; skew 500 ppm
I20260926 13:14:27.465372 5333 webserver.cc:460] Webserver started at http://127.0.98.3:41685/ using document root <none> and password file <none>
I20260926 13:14:27.465653 5333 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:27.465728 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:27.465881 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:27.467077 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2e124b351064a37b651ab37f272cbb4"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "f1435a83ec74c7d195ec86093552ff07"
server_key_iv: "0cbb5d62a3f6ed18d433fc9b48425a95"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.467463 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2e124b351064a37b651ab37f272cbb4"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "f1435a83ec74c7d195ec86093552ff07"
server_key_iv: "0cbb5d62a3f6ed18d433fc9b48425a95"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.468876 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:14:27.469769 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.469966 5333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:27.470088 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "e2e124b351064a37b651ab37f272cbb4"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "f1435a83ec74c7d195ec86093552ff07"
server_key_iv: "0cbb5d62a3f6ed18d433fc9b48425a95"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.470213 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:27.474202 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:27.474543 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:27.474727 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:27.474974 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:27.475329 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:27.475394 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.475441 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:27.475476 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.482630 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36085
I20260926 13:14:27.482664 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36085 every 8 connection(s)
I20260926 13:14:27.483107 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:27.486480 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.483958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36235 (local address 127.0.98.62:41291)
0926 13:14:27.484033 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.484035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.484297 (+ 262us) server_negotiation.cc:407] Connection header received
0926 13:14:27.484775 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.484776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.484782 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.484861 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.486069 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.486071 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.486071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.486120 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.486145 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.486151 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.486294 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.486405 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.486422 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 13:14:27.486618 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.483938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41291 (local address 127.0.98.3:36235)
0926 13:14:27.484154 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.484198 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.484673 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.484923 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.484927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.484935 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.485901 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.485901 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.485932 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.485960 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.486296 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.486301 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.486304 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.486371 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.486424 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":121,"threads_started":1}
I20260926 13:14:27.487092 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41291
I20260926 13:14:27.487185 5461 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:27.487351 5461 heartbeater.cc:511] Master 127.0.98.62:41291 requested a full tablet report, sending...
I20260926 13:14:27.487639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5333
I20260926 13:14:27.487779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:27.487761 5011 ts_manager.cc:194] Registered new tserver with Master: e2e124b351064a37b651ab37f272cbb4 (127.0.98.3:36085)
I20260926 13:14:27.488045 392 external_mini_cluster.cc:1428] Setting key db6970a9c65eedfbbfc6ac231f78d52d
I20260926 13:14:27.489054 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:14:27.491791 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.490698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.98.62:41291)
0926 13:14:27.490747 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.490749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.490757 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:14:27.490842 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.490842 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.490847 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.490901 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.491542 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.491544 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.491544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.491548 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.491566 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.491570 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.491638 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.491696 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.491714 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:27.493860 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:27.495510 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:27.499686 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.497873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36407 (local address 127.0.0.1:60656)
0926 13:14:27.498232 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.498242 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.498288 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.498624 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.498631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.498638 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.499189 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.499190 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.499208 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.499232 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.499460 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.499468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.499472 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.499572 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.499592 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":4,"spinlock_wait_cycles":896,"thread_start_us":89,"threads_started":1}
I20260926 13:14:27.499707 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.498272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.0.98.1:36407)
0926 13:14:27.498464 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.498469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.498479 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:27.498508 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.498510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.498521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.498849 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.499313 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.499319 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.499319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.499324 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.499357 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.499376 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.499441 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.499587 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.499604 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260926 13:14:27.499867 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.498005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.0.98.2:33517)
0926 13:14:27.498600 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.498604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.498621 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:14:27.498648 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.498651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.498746 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.498922 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.499476 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.499484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.499484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.499488 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.499526 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.499565 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.499728 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.499749 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.499764 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":134,"threads_started":1}
I20260926 13:14:27.500023 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.497873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33517 (local address 127.0.0.1:37460)
0926 13:14:27.498044 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.498094 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.498197 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.498943 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.498944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.498946 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.499377 (+ 431us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.499378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.499384 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.499394 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.499644 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.499645 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.499646 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.499675 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.499688 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260926 13:14:27.500783 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.498075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36085 (local address 127.0.0.1:39390)
0926 13:14:27.498412 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.498422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.498472 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.499396 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.499397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.499399 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.500125 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.500126 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.500134 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.500149 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.500622 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.500623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.500623 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.500690 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.500706 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":77,"threads_started":1}
I20260926 13:14:27.500824 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 48c3f51ada4145c1a518c80ca155b7b1 (DEFAULT_TABLE table=test-table [id=406280cd16374ad296c59c7190c0e0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:27.501080 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.498214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39390 (local address 127.0.98.3:36085)
0926 13:14:27.498423 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.498429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.498479 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:14:27.498536 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.498539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.498551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.498660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.500237 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.500245 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.500245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.500251 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.500290 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.500316 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.500406 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.500957 (+ 551us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.500978 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260926 13:14:27.501034 5133 tablet_service.cc:1467] Processing CreateTablet for tablet 48c3f51ada4145c1a518c80ca155b7b1 (DEFAULT_TABLE table=test-table [id=406280cd16374ad296c59c7190c0e0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:27.501251 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c3f51ada4145c1a518c80ca155b7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:27.501451 5133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c3f51ada4145c1a518c80ca155b7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:27.501688 5395 tablet_service.cc:1467] Processing CreateTablet for tablet 48c3f51ada4145c1a518c80ca155b7b1 (DEFAULT_TABLE table=test-table [id=406280cd16374ad296c59c7190c0e0fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:27.502008 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c3f51ada4145c1a518c80ca155b7b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:27.504439 5475 tablet_bootstrap.cc:492] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: Bootstrap starting.
I20260926 13:14:27.505163 5477 tablet_bootstrap.cc:492] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Bootstrap starting.
I20260926 13:14:27.505349 5475 tablet_bootstrap.cc:654] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:27.505383 5476 tablet_bootstrap.cc:492] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Bootstrap starting.
I20260926 13:14:27.505728 5475 log.cc:824] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:27.505833 5477 tablet_bootstrap.cc:654] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:27.506007 5476 tablet_bootstrap.cc:654] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:27.506206 5477 log.cc:824] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:27.506330 5476 log.cc:824] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:27.506716 5475 tablet_bootstrap.cc:492] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: No bootstrap required, opened a new log
I20260926 13:14:27.506809 5475 ts_tablet_manager.cc:1397] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:14:27.506986 5477 tablet_bootstrap.cc:492] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: No bootstrap required, opened a new log
I20260926 13:14:27.507064 5477 ts_tablet_manager.cc:1397] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:27.507622 5476 tablet_bootstrap.cc:492] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: No bootstrap required, opened a new log
I20260926 13:14:27.507710 5476 ts_tablet_manager.cc:1397] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:27.508991 5477 raft_consensus.cc:348] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.509166 5477 raft_consensus.cc:374] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:27.509212 5477 raft_consensus.cc:729] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2e124b351064a37b651ab37f272cbb4, State: Initialized, Role: FOLLOWER
I20260926 13:14:27.509332 5477 consensus_queue.cc:260] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.509619 5477 ts_tablet_manager.cc:1428] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:14:27.509629 5461 heartbeater.cc:503] Master 127.0.98.62:41291 was elected leader, sending a full tablet report...
I20260926 13:14:27.509933 5475 raft_consensus.cc:348] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.510128 5475 raft_consensus.cc:374] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:27.510236 5475 raft_consensus.cc:729] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60027dbfdaa14a37a8d865f57cf8da08, State: Initialized, Role: FOLLOWER
I20260926 13:14:27.510382 5475 consensus_queue.cc:260] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.510756 5199 heartbeater.cc:503] Master 127.0.98.62:41291 was elected leader, sending a full tablet report...
I20260926 13:14:27.511040 5475 ts_tablet_manager.cc:1428] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260926 13:14:27.511139 5476 raft_consensus.cc:348] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.511301 5476 raft_consensus.cc:374] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:27.511379 5476 raft_consensus.cc:729] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57dc8b4c43744edbbf935381c77bb3b, State: Initialized, Role: FOLLOWER
I20260926 13:14:27.511510 5476 consensus_queue.cc:260] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.511852 5476 ts_tablet_manager.cc:1428] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260926 13:14:27.511861 5330 heartbeater.cc:503] Master 127.0.98.62:41291 was elected leader, sending a full tablet report...
I20260926 13:14:27.547379 5482 raft_consensus.cc:482] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:27.547510 5482 raft_consensus.cc:504] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.548019 5482 leader_election.cc:288] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a57dc8b4c43744edbbf935381c77bb3b (127.0.98.2:33517), e2e124b351064a37b651ab37f272cbb4 (127.0.98.3:36085)
I20260926 13:14:27.549575 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.548268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33517 (local address 127.0.98.1:38663)
0926 13:14:27.548330 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.548344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.548411 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.548619 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.548620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.548622 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.549125 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.549125 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.549134 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.549148 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.549396 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.549397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.549397 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.549475 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.549490 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:14:27.549609 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.548327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38663 (local address 127.0.98.2:33517)
0926 13:14:27.548380 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.548382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.548399 (+ 17us) server_negotiation.cc:407] Connection header received
0926 13:14:27.548501 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.548502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.548509 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.548560 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.549278 (+ 718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.549280 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.549280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.549285 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.549306 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.549311 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.549393 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.549496 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.549510 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:14:27.549747 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.548403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36085 (local address 127.0.98.1:51237)
0926 13:14:27.548585 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.548595 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.548681 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.548853 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.548854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.548856 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:27.549329 (+ 473us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:14:27.549329 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:27.549337 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:14:27.549350 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:27.549557 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:27.549558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:27.549558 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 13:14:27.549641 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:14:27.549656 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260926 13:14:27.549753 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.548526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51237 (local address 127.0.98.3:36085)
0926 13:14:27.548573 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.548575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.548669 (+ 94us) server_negotiation.cc:407] Connection header received
0926 13:14:27.548754 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.548755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.548761 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.548834 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.549456 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.549458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.549459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.549464 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.549484 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.549489 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.549546 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.549651 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.549666 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:27.549933 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3f51ada4145c1a518c80ca155b7b1" candidate_uuid: "60027dbfdaa14a37a8d865f57cf8da08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57dc8b4c43744edbbf935381c77bb3b" is_pre_election: true
I20260926 13:14:27.550132 5284 raft_consensus.cc:2454] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60027dbfdaa14a37a8d865f57cf8da08 in term 0.
I20260926 13:14:27.550284 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3f51ada4145c1a518c80ca155b7b1" candidate_uuid: "60027dbfdaa14a37a8d865f57cf8da08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e124b351064a37b651ab37f272cbb4" is_pre_election: true
I20260926 13:14:27.550426 5086 leader_election.cc:302] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60027dbfdaa14a37a8d865f57cf8da08, a57dc8b4c43744edbbf935381c77bb3b; no voters:
I20260926 13:14:27.550505 5415 raft_consensus.cc:2454] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60027dbfdaa14a37a8d865f57cf8da08 in term 0.
I20260926 13:14:27.550606 5482 raft_consensus.cc:2781] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:27.550693 5482 raft_consensus.cc:482] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:27.550732 5482 raft_consensus.cc:3037] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:27.551681 5482 raft_consensus.cc:504] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.551852 5482 leader_election.cc:288] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [CANDIDATE]: Term 1 election: Requested vote from peers a57dc8b4c43744edbbf935381c77bb3b (127.0.98.2:33517), e2e124b351064a37b651ab37f272cbb4 (127.0.98.3:36085)
I20260926 13:14:27.552070 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3f51ada4145c1a518c80ca155b7b1" candidate_uuid: "60027dbfdaa14a37a8d865f57cf8da08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2e124b351064a37b651ab37f272cbb4"
I20260926 13:14:27.552095 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c3f51ada4145c1a518c80ca155b7b1" candidate_uuid: "60027dbfdaa14a37a8d865f57cf8da08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a57dc8b4c43744edbbf935381c77bb3b"
I20260926 13:14:27.552167 5415 raft_consensus.cc:3037] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:27.552205 5284 raft_consensus.cc:3037] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:27.553054 5415 raft_consensus.cc:2454] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60027dbfdaa14a37a8d865f57cf8da08 in term 1.
I20260926 13:14:27.553054 5284 raft_consensus.cc:2454] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60027dbfdaa14a37a8d865f57cf8da08 in term 1.
I20260926 13:14:27.553243 5086 leader_election.cc:302] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60027dbfdaa14a37a8d865f57cf8da08, e2e124b351064a37b651ab37f272cbb4; no voters:
I20260926 13:14:27.553378 5482 raft_consensus.cc:2781] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:27.553555 5482 raft_consensus.cc:686] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 1 LEADER]: Becoming Leader. State: Replica: 60027dbfdaa14a37a8d865f57cf8da08, State: Running, Role: LEADER
I20260926 13:14:27.553668 5482 consensus_queue.cc:237] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } }
I20260926 13:14:27.554695 5010 catalog_manager.cc:5495] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60027dbfdaa14a37a8d865f57cf8da08 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "60027dbfdaa14a37a8d865f57cf8da08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60027dbfdaa14a37a8d865f57cf8da08" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 } health_report { overall_health: UNKNOWN } } }
I20260926 13:14:27.563551 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.562565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.0.98.1:36407)
0926 13:14:27.562612 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.562615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.562625 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:27.562696 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.562696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.562709 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.562764 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.563326 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.563328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.563328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.563332 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.563347 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.563351 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.563403 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.563479 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.563492 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:14:27.565608 5284 raft_consensus.cc:1261] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 1 FOLLOWER]: Refusing update from remote peer 60027dbfdaa14a37a8d865f57cf8da08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:27.565620 5415 raft_consensus.cc:1261] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 1 FOLLOWER]: Refusing update from remote peer 60027dbfdaa14a37a8d865f57cf8da08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:27.565948 5482 consensus_queue.cc:1035] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57dc8b4c43744edbbf935381c77bb3b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:27.566089 5485 consensus_queue.cc:1035] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2e124b351064a37b651ab37f272cbb4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:27.569355 5490 mvcc.cc:205] Tried to move back new op lower bound from 7333595003145326592 to 7333595003100631040. Current Snapshot: MvccSnapshot[applied={T|T < 7333595003145326592}]
I20260926 13:14:27.569684 5489 mvcc.cc:205] Tried to move back new op lower bound from 7333595003145326592 to 7333595003100631040. Current Snapshot: MvccSnapshot[applied={T|T < 7333595003145326592}]
I20260926 13:14:27.571298 5010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:27.571401 5010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:27.572175 5010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d3d1e3f810449a3bfa5a53f57893d1e: Sending DeleteTablet for 3 replicas of tablet 48c3f51ada4145c1a518c80ca155b7b1
I20260926 13:14:27.572505 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c3f51ada4145c1a518c80ca155b7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:27 UTC) from {username='slave'} at 127.0.0.1:37460
I20260926 13:14:27.572551 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c3f51ada4145c1a518c80ca155b7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:27 UTC) from {username='slave'} at 127.0.0.1:39390
I20260926 13:14:27.572543 5133 tablet_service.cc:1514] Processing DeleteTablet for tablet 48c3f51ada4145c1a518c80ca155b7b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:27 UTC) from {username='slave'} at 127.0.0.1:60656
I20260926 13:14:27.572808 5500 tablet_replica.cc:330] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08: stopping tablet replica
I20260926 13:14:27.572919 5500 raft_consensus.cc:2229] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:27.573004 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5071
I20260926 13:14:27.573038 5500 raft_consensus.cc:2258] T 48c3f51ada4145c1a518c80ca155b7b1 P 60027dbfdaa14a37a8d865f57cf8da08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:27.573568 5498 tablet_replica.cc:330] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: stopping tablet replica
I20260926 13:14:27.573678 5498 raft_consensus.cc:2229] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:27.573753 5498 raft_consensus.cc:2258] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:27.573947 5499 tablet_replica.cc:330] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: stopping tablet replica
I20260926 13:14:27.574040 5499 raft_consensus.cc:2229] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:27.574141 5499 raft_consensus.cc:2258] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:27.574314 5498 ts_tablet_manager.cc:1905] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:27.574623 5499 ts_tablet_manager.cc:1905] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:27.576339 5498 ts_tablet_manager.cc:1918] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:27.576339 5499 ts_tablet_manager.cc:1918] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:27.576421 5499 log.cc:1196] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/48c3f51ada4145c1a518c80ca155b7b1
I20260926 13:14:27.576421 5498 log.cc:1196] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/48c3f51ada4145c1a518c80ca155b7b1
I20260926 13:14:27.576686 5498 ts_tablet_manager.cc:1939] T 48c3f51ada4145c1a518c80ca155b7b1 P a57dc8b4c43744edbbf935381c77bb3b: Deleting consensus metadata
I20260926 13:14:27.576686 5499 ts_tablet_manager.cc:1939] T 48c3f51ada4145c1a518c80ca155b7b1 P e2e124b351064a37b651ab37f272cbb4: Deleting consensus metadata
I20260926 13:14:27.577159 4995 catalog_manager.cc:4841] TS e2e124b351064a37b651ab37f272cbb4 (127.0.98.3:36085): tablet 48c3f51ada4145c1a518c80ca155b7b1 (table test-table [id=406280cd16374ad296c59c7190c0e0fa]) successfully deleted
I20260926 13:14:27.577283 4995 catalog_manager.cc:4841] TS a57dc8b4c43744edbbf935381c77bb3b (127.0.98.2:33517): tablet 48c3f51ada4145c1a518c80ca155b7b1 (table test-table [id=406280cd16374ad296c59c7190c0e0fa]) successfully deleted
W20260926 13:14:27.579618 4996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36407 (error 108)
I20260926 13:14:27.579922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5202
I20260926 13:14:27.580190 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.580052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36407 (local address 127.0.0.1:60676)
0926 13:14:27.580109 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.580153 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 13:14:27.580312 4996 catalog_manager.cc:4584] TS 60027dbfdaa14a37a8d865f57cf8da08 (127.0.98.1:36407): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48c3f51ada4145c1a518c80ca155b7b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36407: connect: Connection refused (error 111)
I20260926 13:14:27.585603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5333
I20260926 13:14:27.590268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 4980
2026-09-26T13:14:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (670 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 13:14:27.610800 392 mini_oidc.cc:152] Starting JWKS server
I20260926 13:14:27.611879 392 webserver.cc:460] Webserver started at https://127.0.0.1:35269/ using document root <none> and password file <none>
I20260926 13:14:27.612010 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:14:27.612203 392 webserver.cc:460] Webserver started at http://127.0.0.1:43017/ using document root <none> and password file <none>
2026-09-26T13:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:27Z Disabled control of system clock
I20260926 13:14:27.620322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35113
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35113
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35269/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/testchainca.cert with env {}
W20260926 13:14:27.703616 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:27.703861 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:27.703886 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:27.705112 5507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:27.705154 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:27.705173 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:27.705185 5507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:27.705199 5507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:14:27.705210 5507 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:14:27.705221 5507 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35269/jwks.json
I20260926 13:14:27.706712 5507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35269/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:27.706913 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:27.707104 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:27.712106 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.712106 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.712106 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.712138 5507 server_base.cc:1034] running on GCE node
I20260926 13:14:27.712649 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:27.713018 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:27.714258 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790428467714240 us; error 30 us; skew 500 ppm
I20260926 13:14:27.715835 5507 webserver.cc:460] Webserver started at http://127.0.98.62:35711/ using document root <none> and password file <none>
I20260926 13:14:27.716140 5507 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:27.716215 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:27.716364 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:27.717484 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "5e8cd48eb1a948324a0a27941749dd85"
server_key_iv: "2c7ae76bf082d25fa53b50b90ff08d96"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.717864 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "5e8cd48eb1a948324a0a27941749dd85"
server_key_iv: "2c7ae76bf082d25fa53b50b90ff08d96"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.719365 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:27.720257 5521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.720474 5507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:27.720588 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "5e8cd48eb1a948324a0a27941749dd85"
server_key_iv: "2c7ae76bf082d25fa53b50b90ff08d96"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.720688 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:27.743187 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:27.743561 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:27.743732 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:27.748085 5507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35113
I20260926 13:14:27.748162 5573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35113 every 8 connection(s)
I20260926 13:14:27.748533 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:27.749614 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:27.752189 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2: Bootstrap starting.
I20260926 13:14:27.752782 5574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:27.753051 5574 log.cc:824] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:27.753834 5574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2: No bootstrap required, opened a new log
I20260926 13:14:27.755389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5507
I20260926 13:14:27.755511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:27.755685 392 external_mini_cluster.cc:1428] Setting key 74a6fea49b83621860200dbe3d63f7af
I20260926 13:14:27.755739 5574 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35113 } }
I20260926 13:14:27.755990 5574 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:27.756033 5574 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ee43bf63f1488ca63e968fd7a3b3a2, State: Initialized, Role: FOLLOWER
I20260926 13:14:27.756219 5574 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35113 } }
I20260926 13:14:27.756315 5574 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:27.756356 5574 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:27.756402 5574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:27.757234 5574 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35113 } }
I20260926 13:14:27.757365 5574 leader_election.cc:302] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7ee43bf63f1488ca63e968fd7a3b3a2; no voters:
I20260926 13:14:27.757537 5574 leader_election.cc:288] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:27.757572 5579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:27.757774 5579 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [term 1 LEADER]: Becoming Leader. State: Replica: f7ee43bf63f1488ca63e968fd7a3b3a2, State: Running, Role: LEADER
I20260926 13:14:27.757839 5574 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:27.757870 5579 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35113 } }
I20260926 13:14:27.758268 5580 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35113 } } }
I20260926 13:14:27.758368 5580 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:27.758260 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7ee43bf63f1488ca63e968fd7a3b3a2. Latest consensus state: current_term: 1 leader_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7ee43bf63f1488ca63e968fd7a3b3a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35113 } } }
I20260926 13:14:27.758553 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:27.759248 5586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:27.759856 5586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:27.761762 5586 catalog_manager.cc:1347] Generated new cluster ID: 2859125d56eb4d27bd65261cff29105d
I20260926 13:14:27.761821 5586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:27.762814 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.756363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.0.98.62:35113)
0926 13:14:27.756590 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.756610 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.756650 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:14:27.756781 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.756788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.756875 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.757044 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.757267 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.759209 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.760397 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.761035 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.761165 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:27.762081 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.762085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.762085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.762334 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.762368 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.762381 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.762460 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.762538 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.762622 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":101,"threads_started":1}
I20260926 13:14:27.763731 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:14:27.778822 5586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:27.779533 5586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:27.790094 5586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7ee43bf63f1488ca63e968fd7a3b3a2: Generated new TSK 0
I20260926 13:14:27.790449 5586 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:14:27.817875 5604 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 13:14:27.818019 5604 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260926 13:14:27.818670 5603 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:27.801600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35113 (local address 127.0.0.1:54310)
0926 13:14:27.801797 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:27.801807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:27.801874 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:27.802070 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:27.802071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:27.802072 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:14:27.802132 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:27.802134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.803164 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.803165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:27.803762 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:27.803764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.804267 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.804277 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:27.804330 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:27.804351 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:14:27.818188 (+ 13837us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:14:27.818309 (+ 121us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":105,"threads_started":1}
I20260926 13:14:27.819001 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.801694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.0.98.62:35113)
0926 13:14:27.801763 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.801766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.801882 (+ 116us) server_negotiation.cc:407] Connection header received
0926 13:14:27.801956 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.801958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.801971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.802058 (+ 87us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:14:27.802253 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.803046 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.803847 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.804155 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.804232 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:27.804476 (+ 244us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:14:27.817973 (+ 13497us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 13:14:27.818933 (+ 960us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54310: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 13:14:27.819052 5578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54310: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35269/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 13:14:27.819213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5507
2026-09-26T13:14:27Z chronyd exiting
I20260926 13:14:27.855561 392 test_util.cc:175] -----------------------------------------------
I20260926 13:14:27.855652 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790428444292953-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 13:14:27.856596 392 mini_oidc.cc:152] Starting JWKS server
I20260926 13:14:27.857321 392 webserver.cc:460] Webserver started at https://127.0.0.1:44215/ using document root <none> and password file <none>
I20260926 13:14:27.857365 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:14:27.857537 392 webserver.cc:460] Webserver started at http://127.0.0.1:41943/ using document root <none> and password file <none>
2026-09-26T13:14:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:27Z Disabled control of system clock
I20260926 13:14:27.865679 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36591
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36591
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44215/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/testchainca.cert with env {}
W20260926 13:14:27.948598 5610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:27.948841 5610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:27.948865 5610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:27.950016 5610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:27.950050 5610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:27.950063 5610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:27.950075 5610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:27.950088 5610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:14:27.950141 5610 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:14:27.950192 5610 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44215/jwks.json
I20260926 13:14:27.951766 5610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44215/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:27.952026 5610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:27.952263 5610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:27.957305 5618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.957393 5615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:27.957302 5616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:27.957597 5610 server_base.cc:1034] running on GCE node
I20260926 13:14:27.957952 5610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:27.958338 5610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:27.959522 5610 hybrid_clock.cc:648] HybridClock initialized: now 1790428467959508 us; error 32 us; skew 500 ppm
I20260926 13:14:27.960938 5610 webserver.cc:460] Webserver started at http://127.0.98.62:40859/ using document root <none> and password file <none>
I20260926 13:14:27.961230 5610 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:27.961311 5610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:27.961474 5610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:27.962669 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "bfbb4c2423fc4c71a59df0735ef5296d"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "339174a32fec26083aeca1935b16caa4"
server_key_iv: "7fe8a88523423d96effc698f2afa7c20"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.963044 5610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "bfbb4c2423fc4c71a59df0735ef5296d"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "339174a32fec26083aeca1935b16caa4"
server_key_iv: "7fe8a88523423d96effc698f2afa7c20"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.964507 5610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:27.965401 5624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:27.965582 5610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:27.965690 5610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "bfbb4c2423fc4c71a59df0735ef5296d"
format_stamp: "Formatted at 2026-09-26 13:14:27 on dist-test-slave-f864"
server_key: "339174a32fec26083aeca1935b16caa4"
server_key_iv: "7fe8a88523423d96effc698f2afa7c20"
server_key_version: "encryptionkey@0"
I20260926 13:14:27.965781 5610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:27.977421 5610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:27.977810 5610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:27.977988 5610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:27.982028 5610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36591
I20260926 13:14:27.982059 5676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36591 every 8 connection(s)
I20260926 13:14:27.982462 5610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:27.983532 5677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:27.986064 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d: Bootstrap starting.
I20260926 13:14:27.986799 5677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:27.987142 5677 log.cc:824] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:27.987943 5677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d: No bootstrap required, opened a new log
I20260926 13:14:27.989653 5677 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260926 13:14:27.989810 5677 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:27.989835 5677 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbb4c2423fc4c71a59df0735ef5296d, State: Initialized, Role: FOLLOWER
I20260926 13:14:27.990001 5677 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260926 13:14:27.990067 5677 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:27.990110 5677 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:27.990204 5677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:27.990814 5677 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260926 13:14:27.990829 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5610
I20260926 13:14:27.990907 5677 leader_election.cc:302] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfbb4c2423fc4c71a59df0735ef5296d; no voters:
I20260926 13:14:27.990947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:27.991125 392 external_mini_cluster.cc:1428] Setting key 19bb5e8905c60c2210c68bb9713ce08e
I20260926 13:14:27.991127 5677 leader_election.cc:288] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:27.991159 5680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:27.991279 5680 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [term 1 LEADER]: Becoming Leader. State: Replica: bfbb4c2423fc4c71a59df0735ef5296d, State: Running, Role: LEADER
I20260926 13:14:27.991420 5677 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:27.991427 5680 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } }
I20260926 13:14:27.991986 5681 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } } }
I20260926 13:14:27.992362 5681 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:27.992288 5682 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfbb4c2423fc4c71a59df0735ef5296d. Latest consensus state: current_term: 1 leader_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfbb4c2423fc4c71a59df0735ef5296d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36591 } } }
I20260926 13:14:27.992627 5682 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d [sys.catalog]: This master's current role is: LEADER
W20260926 13:14:27.993450 5698 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:27.993507 5698 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:27.993551 5694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:27.993705 5694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:27.995425 5694 catalog_manager.cc:1347] Generated new cluster ID: 427ea6045fd445a2bf32506425634666
I20260926 13:14:27.995487 5694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:27.997344 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:27.991796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.0.98.62:36591)
0926 13:14:27.992041 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:27.992051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:27.992086 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:14:27.992286 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:27.992294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:27.992374 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:27.992770 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:27.992900 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.994025 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.995164 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:27.995722 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:27.995858 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:27.996599 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:27.996603 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:27.996603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:27.996789 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:27.996822 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:27.996834 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:27.996911 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:27.996987 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:14:27.997177 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260926 13:14:27.998312 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:14:28.006007 5694 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:28.006639 5694 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:28.017284 5694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfbb4c2423fc4c71a59df0735ef5296d: Generated new TSK 0
I20260926 13:14:28.017504 5694 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 13:14:28.043329 5707 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 13:14:28.043434 5707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 13:14:28.043774 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:28.028590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.62:36591)
0926 13:14:28.028685 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:28.028687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:28.028827 (+ 140us) server_negotiation.cc:407] Connection header received
0926 13:14:28.028907 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:28.028908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:28.028919 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:28.028981 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:14:28.029142 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.029669 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.030853 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.031072 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.031124 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.031326 (+ 202us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:14:28.043573 (+ 12247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:14:28.043721 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 13:14:28.043855 5685 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 13:14:28.043980 5706 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:28.028533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36591 (local address 127.0.0.1:47842)
0926 13:14:28.028750 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:28.028762 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:28.028833 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:28.029004 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:28.029005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:28.029010 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:14:28.029072 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:28.029075 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.029759 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.029759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:28.030755 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:28.030759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.031149 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.031157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:28.031222 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.031228 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:14:28.043724 (+ 12496us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 13:14:28.043874 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36591: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44215/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":125,"threads_started":1}
I20260926 13:14:28.044382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5610
2026-09-26T13:14:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 13:14:28.082669 392 mini_oidc.cc:152] Starting JWKS server
I20260926 13:14:28.083417 392 webserver.cc:460] Webserver started at https://127.0.0.1:41351/ using document root <none> and password file <none>
I20260926 13:14:28.083463 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 13:14:28.083619 392 webserver.cc:460] Webserver started at http://127.0.0.1:38207/ using document root <none> and password file <none>
2026-09-26T13:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:28Z Disabled control of system clock
I20260926 13:14:28.091588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39149
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39149
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41351/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:14:28.173147 5713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:28.173377 5713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:28.173400 5713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:28.174723 5713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:28.174779 5713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:28.174793 5713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:28.174805 5713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:28.174818 5713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:14:28.174830 5713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:14:28.174841 5713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41351/jwks.json
I20260926 13:14:28.176270 5713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39149
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41351/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:28.176550 5713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:28.176875 5713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:14:28.182077 5713 server_base.cc:1034] running on GCE node
W20260926 13:14:28.182041 5719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:28.182057 5718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:28.182041 5721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:28.182672 5713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:28.183007 5713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:28.184166 5713 hybrid_clock.cc:648] HybridClock initialized: now 1790428468184154 us; error 33 us; skew 500 ppm
I20260926 13:14:28.185776 5713 webserver.cc:460] Webserver started at http://127.0.98.62:36505/ using document root <none> and password file <none>
I20260926 13:14:28.186050 5713 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:28.186124 5713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:28.186317 5713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:28.187464 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "22dbf5c8dd094e3f8fde531f00229481"
format_stamp: "Formatted at 2026-09-26 13:14:28 on dist-test-slave-f864"
server_key: "921016d0d557671a0110b89755e6d925"
server_key_iv: "45904a36bb9d0c034cdcb2d3684b6c20"
server_key_version: "encryptionkey@0"
I20260926 13:14:28.187860 5713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "22dbf5c8dd094e3f8fde531f00229481"
format_stamp: "Formatted at 2026-09-26 13:14:28 on dist-test-slave-f864"
server_key: "921016d0d557671a0110b89755e6d925"
server_key_iv: "45904a36bb9d0c034cdcb2d3684b6c20"
server_key_version: "encryptionkey@0"
I20260926 13:14:28.189388 5713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:14:28.190503 5727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:28.190805 5713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:28.190943 5713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "22dbf5c8dd094e3f8fde531f00229481"
format_stamp: "Formatted at 2026-09-26 13:14:28 on dist-test-slave-f864"
server_key: "921016d0d557671a0110b89755e6d925"
server_key_iv: "45904a36bb9d0c034cdcb2d3684b6c20"
server_key_version: "encryptionkey@0"
I20260926 13:14:28.191037 5713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:28.202034 5713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:28.202441 5713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:28.202603 5713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:28.206813 5713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39149
I20260926 13:14:28.206875 5779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39149 every 8 connection(s)
I20260926 13:14:28.207252 5713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:28.207988 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:28.211010 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481: Bootstrap starting.
I20260926 13:14:28.211663 5780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:28.211931 5780 log.cc:824] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:28.212672 5780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481: No bootstrap required, opened a new log
I20260926 13:14:28.214377 5780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22dbf5c8dd094e3f8fde531f00229481" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39149 } }
I20260926 13:14:28.214520 5780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:28.214542 5780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22dbf5c8dd094e3f8fde531f00229481, State: Initialized, Role: FOLLOWER
I20260926 13:14:28.214650 5780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22dbf5c8dd094e3f8fde531f00229481" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39149 } }
I20260926 13:14:28.214702 5780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:28.214723 5780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:28.214751 5780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:28.215289 5780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22dbf5c8dd094e3f8fde531f00229481" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39149 } }
I20260926 13:14:28.215381 5780 leader_election.cc:302] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22dbf5c8dd094e3f8fde531f00229481; no voters:
I20260926 13:14:28.215512 5780 leader_election.cc:288] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:28.215595 5783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:28.215811 5780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:28.215808 5783 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [term 1 LEADER]: Becoming Leader. State: Replica: 22dbf5c8dd094e3f8fde531f00229481, State: Running, Role: LEADER
I20260926 13:14:28.216050 5783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22dbf5c8dd094e3f8fde531f00229481" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39149 } }
I20260926 13:14:28.216106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5713
I20260926 13:14:28.216187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:28.216372 392 external_mini_cluster.cc:1428] Setting key b83a3cfaff7d4d302b3a92bd7fccf30f
I20260926 13:14:28.216698 5784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22dbf5c8dd094e3f8fde531f00229481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22dbf5c8dd094e3f8fde531f00229481" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39149 } } }
I20260926 13:14:28.216786 5784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:28.216929 5785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22dbf5c8dd094e3f8fde531f00229481. Latest consensus state: current_term: 1 leader_uuid: "22dbf5c8dd094e3f8fde531f00229481" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22dbf5c8dd094e3f8fde531f00229481" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39149 } } }
I20260926 13:14:28.217073 5785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:28.217288 5793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:28.217854 5793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:28.220551 5793 catalog_manager.cc:1347] Generated new cluster ID: bc291c8bf02140eb94a476946cfdc584
I20260926 13:14:28.220618 5793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:28.223023 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:28.216827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.0.98.62:39149)
0926 13:14:28.217905 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:28.217915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:28.217951 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:14:28.218082 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:28.218090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:28.218210 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:28.218373 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:28.218598 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.219512 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.220670 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.221326 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.221472 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.222265 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:28.222269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:28.222269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:28.222491 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:28.222534 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:28.222556 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:28.222629 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:28.222705 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:14:28.222800 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":75,"threads_started":1}
I20260926 13:14:28.224020 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:14:28.230501 5793 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:28.231241 5793 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:28.238941 5793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22dbf5c8dd094e3f8fde531f00229481: Generated new TSK 0
I20260926 13:14:28.239182 5793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:14:28.253628 5794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:28.249465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.98.62:39149)
0926 13:14:28.249541 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:28.249543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:28.249727 (+ 184us) server_negotiation.cc:407] Connection header received
0926 13:14:28.249774 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:28.249775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:28.249787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:28.249863 (+ 76us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:14:28.250057 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.250692 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.251865 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.252066 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.252132 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.252366 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:14:28.253357 (+ 991us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:14:28.253521 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 13:14:28.253706 5794 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 13:14:28.253714 5809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:28.249402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39149 (local address 127.0.0.1:35250)
0926 13:14:28.249586 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:28.249599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:28.249678 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:28.249884 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:28.249884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:28.249886 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:14:28.249949 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:28.249952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.250783 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.250784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:28.251748 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:28.251751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.252156 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.252163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:28.252224 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.252231 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:14:28.253470 (+ 1239us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 13:14:28.253569 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39149: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41351/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260926 13:14:28.253904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5713
E20260926 13:14:28.266013 5810 webserver.cc:533] Webserver: sslize failed:
E20260926 13:14:28.266103 5810 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T13:14:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (209 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T13:14:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:28Z Disabled control of system clock
I20260926 13:14:28.299947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34635
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34635
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/testchainca.cert with env {}
W20260926 13:14:28.383733 5814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:28.383991 5814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:28.384017 5814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:28.385211 5814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:28.385257 5814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:28.385270 5814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:28.385283 5814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:28.385295 5814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 13:14:28.385308 5814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 13:14:28.385319 5814 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 13:14:28.387995 5814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34635
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:28.388330 5814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:28.388597 5814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:28.395012 5819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:28.395200 5814 server_base.cc:1034] running on GCE node
W20260926 13:14:28.395491 5820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:28.395483 5822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:28.395961 5814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:28.396302 5814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:28.397480 5814 hybrid_clock.cc:648] HybridClock initialized: now 1790428468397467 us; error 31 us; skew 500 ppm
I20260926 13:14:28.398866 5814 webserver.cc:460] Webserver started at http://127.0.98.62:34361/ using document root <none> and password file <none>
I20260926 13:14:28.399133 5814 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:28.399235 5814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:28.399385 5814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:28.400559 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "62b909a3f31a4ea7811391bc081ab344"
format_stamp: "Formatted at 2026-09-26 13:14:28 on dist-test-slave-f864"
server_key: "8ee7763b0aa515a3929c7922289831f7"
server_key_iv: "63f26c8e3aeccef1e4d3701dfbe78098"
server_key_version: "encryptionkey@0"
I20260926 13:14:28.400944 5814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "62b909a3f31a4ea7811391bc081ab344"
format_stamp: "Formatted at 2026-09-26 13:14:28 on dist-test-slave-f864"
server_key: "8ee7763b0aa515a3929c7922289831f7"
server_key_iv: "63f26c8e3aeccef1e4d3701dfbe78098"
server_key_version: "encryptionkey@0"
I20260926 13:14:28.402422 5814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:14:28.403250 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:28.403415 5814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:28.403522 5814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "62b909a3f31a4ea7811391bc081ab344"
format_stamp: "Formatted at 2026-09-26 13:14:28 on dist-test-slave-f864"
server_key: "8ee7763b0aa515a3929c7922289831f7"
server_key_iv: "63f26c8e3aeccef1e4d3701dfbe78098"
server_key_version: "encryptionkey@0"
I20260926 13:14:28.403599 5814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:28.437685 5814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:28.438046 5814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:28.438242 5814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:28.442315 5814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34635
I20260926 13:14:28.442348 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34635 every 8 connection(s)
I20260926 13:14:28.442695 5814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:28.443608 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:28.444921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5814
I20260926 13:14:28.445053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:28.445279 392 external_mini_cluster.cc:1428] Setting key a4cd5c11208f3f89b8b6530802b21bdd
I20260926 13:14:28.446238 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344: Bootstrap starting.
I20260926 13:14:28.446925 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:28.447259 5881 log.cc:824] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:28.447983 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344: No bootstrap required, opened a new log
I20260926 13:14:28.450453 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b909a3f31a4ea7811391bc081ab344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34635 } }
I20260926 13:14:28.450656 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:28.450701 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62b909a3f31a4ea7811391bc081ab344, State: Initialized, Role: FOLLOWER
I20260926 13:14:28.450838 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b909a3f31a4ea7811391bc081ab344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34635 } }
I20260926 13:14:28.450909 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:28.450932 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:28.450997 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:28.451501 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:28.445863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.0.98.62:34635)
0926 13:14:28.446088 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:28.446098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:28.446131 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:14:28.446717 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:28.446726 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:28.446805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:28.446964 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:28.447222 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.448312 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.449428 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.449840 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.449938 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.450701 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:28.450706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:28.450706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:14:28.450955 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:28.451000 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:14:28.451014 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:28.451076 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:28.451196 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 13:14:28.451293 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260926 13:14:28.451676 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b909a3f31a4ea7811391bc081ab344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34635 } }
I20260926 13:14:28.451809 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62b909a3f31a4ea7811391bc081ab344; no voters:
I20260926 13:14:28.451998 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:28.452024 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:28.452169 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [term 1 LEADER]: Becoming Leader. State: Replica: 62b909a3f31a4ea7811391bc081ab344, State: Running, Role: LEADER
I20260926 13:14:28.452252 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b909a3f31a4ea7811391bc081ab344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34635 } }
I20260926 13:14:28.452313 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:28.452387 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:14:28.452548 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62b909a3f31a4ea7811391bc081ab344. Latest consensus state: current_term: 1 leader_uuid: "62b909a3f31a4ea7811391bc081ab344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b909a3f31a4ea7811391bc081ab344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34635 } } }
I20260926 13:14:28.452683 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:28.453374 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62b909a3f31a4ea7811391bc081ab344" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62b909a3f31a4ea7811391bc081ab344" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34635 } } }
I20260926 13:14:28.453492 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344 [sys.catalog]: This master's current role is: LEADER
W20260926 13:14:28.454075 5902 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:28.454120 5902 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:28.454191 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:28.454360 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:28.455926 5892 catalog_manager.cc:1347] Generated new cluster ID: 2ab2e31119434a9aa65056dc920e2128
I20260926 13:14:28.455982 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:28.469904 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:28.470700 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:28.477747 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62b909a3f31a4ea7811391bc081ab344: Generated new TSK 0
I20260926 13:14:28.477967 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:14:28.564345 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:28.485642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.0.98.62:34635)
0926 13:14:28.485718 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:28.485720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:28.485823 (+ 103us) server_negotiation.cc:407] Connection header received
0926 13:14:28.485900 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:28.485900 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:28.485911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:28.485974 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0926 13:14:28.486106 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.486690 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.487899 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.488120 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.488190 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.488389 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 13:14:28.564105 (+ 75716us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:14:28.564259 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37628: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 13:14:28.564442 5884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37628: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 13:14:28.564559 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:28.485567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34635 (local address 127.0.0.1:37628)
0926 13:14:28.485744 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:28.485757 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:28.485829 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:28.485992 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:28.485993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:28.485995 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 13:14:28.486054 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:28.486056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.486777 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.486778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:28.487748 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:28.487751 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:28.488203 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:28.488210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:28.488272 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:28.488278 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 13:14:28.564286 (+ 76008us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 13:14:28.564455 (+ 169us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34635: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":127,"threads_started":1}
I20260926 13:14:28.564963 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5814
2026-09-26T13:14:28Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (290 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:28 dist-test-slave-f864 krb5kdc[5922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:28 dist-test-slave-f864 krb5kdc[5922](info): set up 2 sockets
Sep 26 13:14:28 dist-test-slave-f864 krb5kdc[5922](info): commencing operation
krb5kdc: starting...
W20260926 13:14:29.786527 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:29 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:14:29.915879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44651
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:30.000583 5935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:30.000854 5935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:30.000877 5935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:30.002110 5935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:30.002152 5935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:30.002195 5935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:30.002208 5935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:30.002220 5935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:30.003793 5935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:30.004067 5935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:30.004308 5935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:30.009246 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.009341 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.009222 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:30.009752 5935 server_base.cc:1034] running on GCE node
I20260926 13:14:30.010030 5935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:30.010382 5935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:30.011555 5935 hybrid_clock.cc:648] HybridClock initialized: now 1790428470011542 us; error 48 us; skew 500 ppm
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.014148 5935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:14:30.014724 5935 webserver.cc:460] Webserver started at http://127.0.98.62:46429/ using document root <none> and password file <none>
I20260926 13:14:30.014955 5935 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:30.015024 5935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:30.015177 5935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:30.016321 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "a932480d8c6f4e3aad8436748dadcce8"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "e2ce83af270e754ce3703df4fd504544"
server_key_iv: "3bf07b0407f05786acd73a88af867781"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.016698 5935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "a932480d8c6f4e3aad8436748dadcce8"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "e2ce83af270e754ce3703df4fd504544"
server_key_iv: "3bf07b0407f05786acd73a88af867781"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.018139 5935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 13:14:30.019098 5950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.019424 5935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:30.019529 5935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "a932480d8c6f4e3aad8436748dadcce8"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "e2ce83af270e754ce3703df4fd504544"
server_key_iv: "3bf07b0407f05786acd73a88af867781"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.019621 5935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:30.056447 5935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:30.058882 5935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:30.059075 5935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:30.063416 5935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44651
I20260926 13:14:30.063453 6002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44651 every 8 connection(s)
I20260926 13:14:30.064013 5935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:30.064728 6003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:30.067334 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8: Bootstrap starting.
I20260926 13:14:30.067950 6003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:30.068261 6003 log.cc:824] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:30.069141 6003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8: No bootstrap required, opened a new log
I20260926 13:14:30.070997 6003 raft_consensus.cc:348] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a932480d8c6f4e3aad8436748dadcce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260926 13:14:30.071161 6003 raft_consensus.cc:374] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:30.071183 6003 raft_consensus.cc:729] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a932480d8c6f4e3aad8436748dadcce8, State: Initialized, Role: FOLLOWER
I20260926 13:14:30.071364 6003 consensus_queue.cc:260] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a932480d8c6f4e3aad8436748dadcce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260926 13:14:30.071447 6003 raft_consensus.cc:388] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:30.071470 6003 raft_consensus.cc:482] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:30.071521 6003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:30.071759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 5935
I20260926 13:14:30.071878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:30.072073 392 external_mini_cluster.cc:1428] Setting key c8e4a9850d245f66c95a17ded77a6f6e
I20260926 13:14:30.072913 6003 raft_consensus.cc:504] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a932480d8c6f4e3aad8436748dadcce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260926 13:14:30.073036 6003 leader_election.cc:302] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a932480d8c6f4e3aad8436748dadcce8; no voters:
I20260926 13:14:30.073192 6003 leader_election.cc:288] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:30.073266 6008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:30.073458 6008 raft_consensus.cc:686] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [term 1 LEADER]: Becoming Leader. State: Replica: a932480d8c6f4e3aad8436748dadcce8, State: Running, Role: LEADER
I20260926 13:14:30.073472 6003 sys_catalog.cc:564] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:30.074362 6008 consensus_queue.cc:237] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a932480d8c6f4e3aad8436748dadcce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } }
I20260926 13:14:30.075070 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a932480d8c6f4e3aad8436748dadcce8. Latest consensus state: current_term: 1 leader_uuid: "a932480d8c6f4e3aad8436748dadcce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a932480d8c6f4e3aad8436748dadcce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } } }
I20260926 13:14:30.075166 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:30.075556 6023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:30.075528 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a932480d8c6f4e3aad8436748dadcce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a932480d8c6f4e3aad8436748dadcce8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44651 } } }
I20260926 13:14:30.075626 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:30.076073 6023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:30.078792 6023 catalog_manager.cc:1347] Generated new cluster ID: 91a53cff55e544d09da8d9880d8ed417
I20260926 13:14:30.078855 6023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:30.083671 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.072681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59092 (local address 127.0.98.62:44651)
0926 13:14:30.072916 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.072927 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.072968 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:14:30.073098 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.073106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.073175 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.073337 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.074229 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.075884 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.076761 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.077277 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.077436 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.080218 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.080222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.080223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.080236 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.082031 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.082477 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.082481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.082481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.082590 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.082817 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.082818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.082818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.083083 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.083269 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.083348 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.083499 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":116,"threads_started":1}
I20260926 13:14:30.102540 6023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:30.103338 6023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:30.112732 6023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a932480d8c6f4e3aad8436748dadcce8: Generated new TSK 0
I20260926 13:14:30.112963 6023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:14:30.151659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:30.234422 6029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:30.234650 6029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:30.234674 6029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:30.235870 6029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:30.235921 6029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:30.235949 6029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:30.237462 6029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:30.237725 6029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:30.237986 6029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:30.243176 6034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.243187 6037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.243243 6035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:30.243556 6029 server_base.cc:1034] running on GCE node
I20260926 13:14:30.243850 6029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:30.244159 6029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:30.245316 6029 hybrid_clock.cc:648] HybridClock initialized: now 1790428470245293 us; error 44 us; skew 500 ppm
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.247982 6029 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:14:30.248471 6029 webserver.cc:460] Webserver started at http://127.0.98.1:44059/ using document root <none> and password file <none>
I20260926 13:14:30.248677 6029 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:30.248723 6029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:30.248826 6029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:30.249841 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bb1e5efce4e42dcafe19683b795c140"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "faff63ea6b4872f297e99a072d29eab7"
server_key_iv: "9e201ec941c8e8852f234b05f0d3141a"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.250252 6029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bb1e5efce4e42dcafe19683b795c140"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "faff63ea6b4872f297e99a072d29eab7"
server_key_iv: "9e201ec941c8e8852f234b05f0d3141a"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.251592 6029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:30.252409 6044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.252599 6029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:30.252707 6029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "1bb1e5efce4e42dcafe19683b795c140"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "faff63ea6b4872f297e99a072d29eab7"
server_key_iv: "9e201ec941c8e8852f234b05f0d3141a"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.252789 6029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:30.266907 6029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:30.269408 6029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:30.269586 6029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:30.269850 6029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:30.270247 6029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:30.270304 6029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.270345 6029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:30.270373 6029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.277273 6029 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39733
I20260926 13:14:30.277331 6157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39733 every 8 connection(s)
I20260926 13:14:30.277732 6029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:30.287024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6029
I20260926 13:14:30.287158 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:30.287380 392 external_mini_cluster.cc:1428] Setting key d0d549c0416258d8bdc3b02d0703c09d
I20260926 13:14:30.288553 6160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.278574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.1:41153)
0926 13:14:30.278737 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:30.278789 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:30.279831 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:30.280120 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:30.280123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:30.280461 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:30.280678 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:30.280683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.281433 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.281434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:30.282386 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:30.282389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.282831 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.282833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:30.282948 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.283574 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:30.283583 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:30.285005 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:30.286804 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:30.286811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:30.286814 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:30.287079 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:30.287419 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:30.287420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:30.287421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:30.287595 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:30.287933 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:30.287939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:30.288116 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 13:14:30.288231 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 13:14:30.288377 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260926 13:14:30.289067 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.278663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41153 (local address 127.0.98.62:44651)
0926 13:14:30.278772 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.278775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.278889 (+ 114us) server_negotiation.cc:407] Connection header received
0926 13:14:30.279967 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.279968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.279993 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.280053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.280808 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.281332 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.282489 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.282714 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.282786 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.285124 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.285126 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.285126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.285134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.286695 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.287261 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.287262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.287263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.287328 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.287686 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.287687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.287687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.287870 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.287944 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.288925 (+ 981us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.289006 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 13:14:30.289608 6158 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44651
I20260926 13:14:30.289731 6158 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:30.289948 6158 heartbeater.cc:511] Master 127.0.98.62:44651 requested a full tablet report, sending...
I20260926 13:14:30.290601 5967 ts_manager.cc:194] Registered new tserver with Master: 1bb1e5efce4e42dcafe19683b795c140 (127.0.98.1:39733)
I20260926 13:14:30.291234 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41153
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:14:30.317402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:30.402555 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:30.402778 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:30.402803 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:30.403988 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:30.404034 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:30.404062 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:30.405573 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:30.405849 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:30.406086 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:30.411155 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.411152 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.411161 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:30.411517 6163 server_base.cc:1034] running on GCE node
I20260926 13:14:30.411794 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:30.412079 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:30.413219 6163 hybrid_clock.cc:648] HybridClock initialized: now 1790428470413202 us; error 35 us; skew 500 ppm
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.416126 6163 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:14:30.416633 6163 webserver.cc:460] Webserver started at http://127.0.98.2:38791/ using document root <none> and password file <none>
I20260926 13:14:30.416882 6163 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:30.416952 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:30.417112 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:30.418265 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "985838c2752040569a55825ba34b39b5"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "e676d39d99575a6a5b10bb2e54e037c9"
server_key_iv: "a84b49788dda8f073edd77ad4d26720f"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.418633 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "985838c2752040569a55825ba34b39b5"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "e676d39d99575a6a5b10bb2e54e037c9"
server_key_iv: "a84b49788dda8f073edd77ad4d26720f"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.420033 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:14:30.420815 6178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.421058 6163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:30.421190 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "985838c2752040569a55825ba34b39b5"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "e676d39d99575a6a5b10bb2e54e037c9"
server_key_iv: "a84b49788dda8f073edd77ad4d26720f"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.421298 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:30.449625 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:30.452271 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:30.452471 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:30.452730 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:30.453080 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:30.453137 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.453181 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:30.453207 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.459452 6163 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35169
I20260926 13:14:30.459492 6291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35169 every 8 connection(s)
I20260926 13:14:30.459826 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:30.462886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6163
I20260926 13:14:30.462994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:30.463204 392 external_mini_cluster.cc:1428] Setting key cc5cf9b7b37d7040713a91047eca1de3
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:30.471452 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.460657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44863 (local address 127.0.98.62:44651)
0926 13:14:30.460749 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.460751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.460941 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:14:30.461965 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.461967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.461979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.462049 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.462903 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.463666 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.465175 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.465468 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.465571 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.468276 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.468277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.468277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.468285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.469842 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.470400 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.470401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.470401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.470473 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.470801 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.470801 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.470802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.470910 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.470961 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.471312 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.471385 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 13:14:30.471742 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.460615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.2:44863)
0926 13:14:30.460790 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:30.460834 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:30.461830 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:30.462115 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:30.462119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:30.462488 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:30.462789 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:30.462797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.463762 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.463763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:30.465055 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:30.465058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.465583 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.465585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:30.465739 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.466367 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:30.466376 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:30.468155 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:30.469931 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:30.469940 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:30.469943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:30.470302 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:30.470549 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:30.470550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:30.470551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:30.470713 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:30.470978 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:30.470984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:30.471166 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 13:14:30.471280 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 13:14:30.471463 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:14:30.472409 6292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44651
I20260926 13:14:30.472515 6292 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:30.472721 6292 heartbeater.cc:511] Master 127.0.98.62:44651 requested a full tablet report, sending...
I20260926 13:14:30.473155 5967 ts_manager.cc:194] Registered new tserver with Master: 985838c2752040569a55825ba34b39b5 (127.0.98.2:35169)
I20260926 13:14:30.473628 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:14:30.492944 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:30.580005 6297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:30.580284 6297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:30.580331 6297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:30.581700 6297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:30.581773 6297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:30.581825 6297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:30.583600 6297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:30.583915 6297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:30.584152 6297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:30.589771 6303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:30.589838 6297 server_base.cc:1034] running on GCE node
W20260926 13:14:30.589913 6305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:30.589784 6302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:30.590329 6297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:30.590610 6297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:30.591827 6297 hybrid_clock.cc:648] HybridClock initialized: now 1790428470591809 us; error 29 us; skew 500 ppm
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.594879 6297 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:14:30.595383 6297 webserver.cc:460] Webserver started at http://127.0.98.3:33589/ using document root <none> and password file <none>
I20260926 13:14:30.595649 6297 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:30.595734 6297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:30.595896 6297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:30.597021 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "47e5ca1d1305437b8225fe291ae1eaaa"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "bd5a3b97e6cb422ab44fb425985ecb6e"
server_key_iv: "8e03d77266d53823e7e7234c356bb739"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.597424 6297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47e5ca1d1305437b8225fe291ae1eaaa"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "bd5a3b97e6cb422ab44fb425985ecb6e"
server_key_iv: "8e03d77266d53823e7e7234c356bb739"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.599004 6297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:30.599891 6312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.600077 6297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:30.600178 6297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "47e5ca1d1305437b8225fe291ae1eaaa"
format_stamp: "Formatted at 2026-09-26 13:14:30 on dist-test-slave-f864"
server_key: "bd5a3b97e6cb422ab44fb425985ecb6e"
server_key_iv: "8e03d77266d53823e7e7234c356bb739"
server_key_version: "encryptionkey@0"
I20260926 13:14:30.600286 6297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:30.622575 6297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:30.625281 6297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:30.625456 6297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:30.625788 6297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:30.626219 6297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:30.626291 6297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.626346 6297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:30.626381 6297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:30.632709 6297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38453
I20260926 13:14:30.632750 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38453 every 8 connection(s)
I20260926 13:14:30.633121 6297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:30.638482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6297
I20260926 13:14:30.638582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:30.638747 392 external_mini_cluster.cc:1428] Setting key 977011bdcce168009e659e0fb274e144
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:14:30.642912 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.633928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41507 (local address 127.0.98.62:44651)
0926 13:14:30.633989 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.633991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.634239 (+ 248us) server_negotiation.cc:407] Connection header received
0926 13:14:30.635310 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.635311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.635322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.635377 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.636051 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.636584 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.637655 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.637850 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.637924 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.640452 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.640453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.640453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.640460 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.641646 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.642010 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.642011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.642011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.642063 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.642379 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.642380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.642380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.642484 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.642545 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.642790 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.642849 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:14:30.643050 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.633892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44651 (local address 127.0.98.3:41507)
0926 13:14:30.634073 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:30.634112 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:30.635197 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:30.635440 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:30.635444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:30.635717 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:30.635929 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:30.635933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.636678 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.636679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:30.637563 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:30.637565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.637965 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.637966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:30.638074 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.638710 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:30.638721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:30.640375 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:30.641724 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:30.641729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:30.641731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:30.641945 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:30.642121 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:30.642122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:30.642122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:30.642318 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:30.642550 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:30.642554 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:30.642664 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 13:14:30.642749 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:14:30.642863 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260926 13:14:30.643539 6426 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44651
I20260926 13:14:30.643636 6426 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:30.643832 6426 heartbeater.cc:511] Master 127.0.98.62:44651 requested a full tablet report, sending...
I20260926 13:14:30.644196 5967 ts_manager.cc:194] Registered new tserver with Master: 47e5ca1d1305437b8225fe291ae1eaaa (127.0.98.3:38453)
I20260926 13:14:30.644575 5967 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41507
I20260926 13:14:30.650532 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/test-admin.keytab.
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.669493 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 13:14:30.671525 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:30.670627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.0.98.1:39733)
0926 13:14:30.670860 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.670865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.670878 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:14:30.670907 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.670909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.670924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.671001 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.671455 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60478: BlockingRecv error: recv got EOF from 127.0.0.1:60478 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":144,"threads_started":1}
W20260926 13:14:30.671736 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.672960 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.674146 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:30.683677 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.674845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60482 (local address 127.0.98.1:39733)
0926 13:14:30.674915 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.674918 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.674955 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:14:30.675048 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.675049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.675062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.675122 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.675588 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.676432 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.677291 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.677674 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.677786 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.680371 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.680383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.680384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.680393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.682309 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.682743 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.682755 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.682755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.682815 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.683041 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.683043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.683043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.683204 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.683372 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.683453 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.683541 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 13:14:30.685883 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:30.685214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.0.98.1:39733)
0926 13:14:30.685272 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.685274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.685283 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:30.685304 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.685305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.685322 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.685377 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.685825 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60490: BlockingRecv error: recv got EOF from 127.0.0.1:60490 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 13:14:30.685966 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.687106 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.688189 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:30.696482 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.688874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.0.98.1:39733)
0926 13:14:30.688953 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.688955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.688963 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:14:30.688990 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.688991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.689002 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.689085 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.689502 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.690042 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.690909 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.691142 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.691249 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.693532 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.693534 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.693534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.693543 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.695176 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.695581 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.695582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.695583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.695663 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.695892 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.695892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.695893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.696013 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.696130 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.696283 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.696368 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260926 13:14:30.698393 6433 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:14:30.697849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.0.98.1:39733)
0926 13:14:30.697908 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.697910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.697919 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:30.697938 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.697939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.697955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.698003 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.698354 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60506: BlockingRecv error: recv got EOF from 127.0.0.1:60506 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 13:14:30.698602 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790428444292953-392-0/krb5kdc/krb5cc)
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.699719 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:30.700657 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[5922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428470, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:14:30.708870 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:30.701228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60512 (local address 127.0.98.1:39733)
0926 13:14:30.701298 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:30.701301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:30.701392 (+ 91us) server_negotiation.cc:407] Connection header received
0926 13:14:30.701503 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:30.701504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:30.701513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:30.701592 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:30.702009 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.702617 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.703390 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:30.703691 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:30.703789 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:30.706155 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:30.706156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:30.706156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:30.706195 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:30.707604 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.708015 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.708016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.708017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.708083 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:30.708309 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:30.708310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:30.708311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:30.708426 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:30.708554 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:30.708663 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 13:14:30.708746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:14:30.709596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6029
I20260926 13:14:30.714130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6163
I20260926 13:14:30.718921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6297
I20260926 13:14:30.723312 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 5935
2026-09-26T13:14:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1978 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[6446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[6446](info): set up 2 sockets
Sep 26 13:14:30 dist-test-slave-f864 krb5kdc[6446](info): commencing operation
krb5kdc: starting...
W20260926 13:14:31.770596 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:14:31 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:14:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:14:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 13:14:31.897388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45117
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:31.979372 6459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:31.979607 6459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:31.979631 6459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:31.980780 6459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:31.980823 6459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:31.980839 6459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:31.980850 6459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:31.980863 6459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:31.982343 6459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:31.982550 6459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:31.982734 6459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:31.988054 6465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:31.988054 6467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:31.988298 6464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:31.988520 6459 server_base.cc:1034] running on GCE node
I20260926 13:14:31.988827 6459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:31.989245 6459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:31.990430 6459 hybrid_clock.cc:648] HybridClock initialized: now 1790428471990432 us; error 49 us; skew 500 ppm
Sep 26 13:14:31 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.000627 6459 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 13:14:32.001281 6459 webserver.cc:460] Webserver started at http://127.0.98.62:41607/ using document root <none> and password file <none>
I20260926 13:14:32.001570 6459 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.001649 6459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.001825 6459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:32.003116 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "25e06fc9a29a471182200dfd21afc633"
format_stamp: "Formatted at 2026-09-26 13:14:31 on dist-test-slave-f864"
server_key: "e8895b21bff4f196f120d00b39868214"
server_key_iv: "083039b765ff1e4d8c2da9ccc5b57ca6"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.003520 6459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "25e06fc9a29a471182200dfd21afc633"
format_stamp: "Formatted at 2026-09-26 13:14:31 on dist-test-slave-f864"
server_key: "e8895b21bff4f196f120d00b39868214"
server_key_iv: "083039b765ff1e4d8c2da9ccc5b57ca6"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.004985 6459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:14:32.005851 6474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.006074 6459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:32.006223 6459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "25e06fc9a29a471182200dfd21afc633"
format_stamp: "Formatted at 2026-09-26 13:14:31 on dist-test-slave-f864"
server_key: "e8895b21bff4f196f120d00b39868214"
server_key_iv: "083039b765ff1e4d8c2da9ccc5b57ca6"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.006335 6459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.023160 6459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:32.025583 6459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:32.025748 6459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:32.029955 6459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45117
I20260926 13:14:32.030000 6526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45117 every 8 connection(s)
I20260926 13:14:32.030424 6459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:14:32.031383 6527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:32.032711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6459
I20260926 13:14:32.032810 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:14:32.033015 392 external_mini_cluster.cc:1428] Setting key c2a3710b95dedbbcdb0afa2113aca83e
I20260926 13:14:32.034147 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Bootstrap starting.
I20260926 13:14:32.034956 6527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:32.035301 6527 log.cc:824] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:32.036038 6527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: No bootstrap required, opened a new log
I20260926 13:14:32.038429 6527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } }
I20260926 13:14:32.038640 6527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:32.038682 6527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25e06fc9a29a471182200dfd21afc633, State: Initialized, Role: FOLLOWER
I20260926 13:14:32.038834 6527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } }
I20260926 13:14:32.038928 6527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:14:32.038964 6527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:14:32.039021 6527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:32.039839 6527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } }
I20260926 13:14:32.039933 6527 leader_election.cc:302] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25e06fc9a29a471182200dfd21afc633; no voters:
I20260926 13:14:32.040118 6527 leader_election.cc:288] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:14:32.040179 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:32.040356 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 1 LEADER]: Becoming Leader. State: Replica: 25e06fc9a29a471182200dfd21afc633, State: Running, Role: LEADER
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:32.040398 6527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:32.040529 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } }
I20260926 13:14:32.040920 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e06fc9a29a471182200dfd21afc633. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } }
I20260926 13:14:32.041002 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:32.042215 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } }
I20260926 13:14:32.042299 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: This master's current role is: LEADER
W20260926 13:14:32.042907 6547 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:32.042975 6547 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:32.043031 6548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:14:32.043210 6548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:14:32.045027 6548 catalog_manager.cc:1347] Generated new cluster ID: 2d35d30aae284e0eb9f9a552b2cafde8
I20260926 13:14:32.045086 6548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:14:32.045347 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.033590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.0.98.62:45117)
0926 13:14:32.033770 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.033779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.033824 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:14:32.034350 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.034360 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.034422 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.034548 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.035446 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.036443 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.037478 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.037872 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.037954 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.041361 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.041365 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.041366 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.041379 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.043502 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.043982 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.043985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.043985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.044094 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.044366 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.044367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.044367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.044663 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.044848 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.044929 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.045073 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260926 13:14:32.069478 6548 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:14:32.070065 6548 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:14:32.073028 6548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Generated new TSK 0
I20260926 13:14:32.073221 6548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 13:14:32.105799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45117
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:32.188851 6553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:32.189081 6553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:32.189107 6553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:32.190428 6553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:32.190485 6553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:32.190514 6553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:32.192059 6553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:32.192335 6553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:32.192605 6553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:32.197907 6559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.197911 6561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.197988 6553 server_base.cc:1034] running on GCE node
W20260926 13:14:32.197907 6558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.198427 6553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:32.198742 6553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:32.199921 6553 hybrid_clock.cc:648] HybridClock initialized: now 1790428472199898 us; error 39 us; skew 500 ppm
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.202713 6553 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:14:32.203152 6553 webserver.cc:460] Webserver started at http://127.0.98.1:41895/ using document root <none> and password file <none>
I20260926 13:14:32.203409 6553 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.203480 6553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.203629 6553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:32.204733 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "1affc48a249337901d3a25c619f4ce64"
server_key_iv: "fd64cf88bf9556b90227f7e6a17a0626"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.205107 6553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "1affc48a249337901d3a25c619f4ce64"
server_key_iv: "fd64cf88bf9556b90227f7e6a17a0626"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.206508 6553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:32.207257 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.207474 6553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:32.207574 6553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "1affc48a249337901d3a25c619f4ce64"
server_key_iv: "fd64cf88bf9556b90227f7e6a17a0626"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.207659 6553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.215196 6553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:32.217269 6553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:32.217442 6553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:32.217675 6553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:32.217986 6553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:32.218042 6553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.218082 6553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:32.218110 6553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.224000 6553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33261
I20260926 13:14:32.224080 6681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33261 every 8 connection(s)
I20260926 13:14:32.224423 6553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:32.231182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6553
I20260926 13:14:32.231299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:14:32.231539 392 external_mini_cluster.cc:1428] Setting key 30d5eea00eb91dba37100fec33dee44e
I20260926 13:14:32.234339 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.225200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37723 (local address 127.0.98.62:45117)
0926 13:14:32.225246 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.225248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.225500 (+ 252us) server_negotiation.cc:407] Connection header received
0926 13:14:32.226004 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.226005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.226016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.226083 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.227154 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.227696 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.228678 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.228886 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.228969 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.231181 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.231183 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.231183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.231192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.232727 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.233241 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.233242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.233242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.233291 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.233586 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.233586 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.233586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.233731 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.233807 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.234095 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.234213 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:14:32.234437 6684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.225159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:37723)
0926 13:14:32.225360 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.225405 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.225884 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.226158 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.226210 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.226832 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.227031 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.227036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.227789 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.227790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.228589 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.228591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.228983 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.228984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.229078 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.229657 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.229666 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.231073 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.232827 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.232834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.232836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.233164 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.233354 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.233355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.233356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.233504 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.233815 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.233821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.233953 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.234050 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.234234 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260926 13:14:32.234877 6682 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45117
I20260926 13:14:32.234972 6682 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:32.235167 6682 heartbeater.cc:511] Master 127.0.98.62:45117 requested a full tablet report, sending...
I20260926 13:14:32.235668 6491 ts_manager.cc:194] Registered new tserver with Master: bc44d8c907bf4f898a3aa007a9a3b9e5 (127.0.98.1:33261)
I20260926 13:14:32.236274 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37723
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 13:14:32.257642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45117
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:32.338748 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:32.339023 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:32.339062 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:32.340405 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:32.340492 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:32.340540 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:32.342314 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:32.342599 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:32.342851 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:32.347950 6692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.347995 6693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.347980 6695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.348208 6687 server_base.cc:1034] running on GCE node
I20260926 13:14:32.348467 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:32.348760 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:32.349952 6687 hybrid_clock.cc:648] HybridClock initialized: now 1790428472349935 us; error 28 us; skew 500 ppm
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.352597 6687 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:14:32.353039 6687 webserver.cc:460] Webserver started at http://127.0.98.2:37865/ using document root <none> and password file <none>
I20260926 13:14:32.353235 6687 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.353276 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.353364 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:32.354506 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "b290466c423a449b853ba9387ca51b2c"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "8adcd00dec72d4e46615ab1b0afc5592"
server_key_iv: "36e3513f4ddaa07cc38f8366ddd7c453"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.354851 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b290466c423a449b853ba9387ca51b2c"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "8adcd00dec72d4e46615ab1b0afc5592"
server_key_iv: "36e3513f4ddaa07cc38f8366ddd7c453"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.356146 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:32.356878 6702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.357195 6687 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:32.357273 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "b290466c423a449b853ba9387ca51b2c"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "8adcd00dec72d4e46615ab1b0afc5592"
server_key_iv: "36e3513f4ddaa07cc38f8366ddd7c453"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.357375 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.368391 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:32.370877 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:32.371011 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:32.371205 6687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:32.371479 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:32.371511 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.371533 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:32.371546 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.377158 6687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43917
I20260926 13:14:32.377364 6815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43917 every 8 connection(s)
I20260926 13:14:32.377605 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:32.382565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6687
I20260926 13:14:32.382676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:14:32.382906 392 external_mini_cluster.cc:1428] Setting key a0f6fa27c658fece4c3f813120d67fb8
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:32.388494 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.378521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46793 (local address 127.0.98.62:45117)
0926 13:14:32.378587 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.378589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.378769 (+ 180us) server_negotiation.cc:407] Connection header received
0926 13:14:32.379280 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.379281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.379292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.379357 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.380272 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.380773 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.381766 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.381954 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.382008 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.384512 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.384513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.384514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.384521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.386337 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.386973 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.386974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.386975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.387071 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.387425 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.387426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.387427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.387663 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.387765 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.388162 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.388308 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:14:32.388908 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.378459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:46793)
0926 13:14:32.378633 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.378675 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.379188 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.379419 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.379423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.379989 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.380186 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.380190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.380866 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.380867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.381685 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.381687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.382033 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.382034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.382135 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.382673 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.382684 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.384421 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.386467 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.386481 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.386484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.386833 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.387159 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.387160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.387160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.387319 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.387746 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.387754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.387943 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.388107 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.388312 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":103,"threads_started":1}
I20260926 13:14:32.390810 6816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45117
I20260926 13:14:32.390937 6816 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:32.391152 6816 heartbeater.cc:511] Master 127.0.98.62:45117 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:14:32.391697 6491 ts_manager.cc:194] Registered new tserver with Master: b290466c423a449b853ba9387ca51b2c (127.0.98.2:43917)
I20260926 13:14:32.392230 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 13:14:32.410470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45117
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:32.493553 6821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:32.493777 6821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:32.493800 6821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:32.495085 6821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:32.495132 6821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:32.495159 6821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:32.496668 6821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:32.496915 6821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:32.497171 6821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:32.502259 6826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.502252 6829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.502252 6827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.502923 6821 server_base.cc:1034] running on GCE node
I20260926 13:14:32.503144 6821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:32.503401 6821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:32.504549 6821 hybrid_clock.cc:648] HybridClock initialized: now 1790428472504539 us; error 26 us; skew 500 ppm
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.507274 6821 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:14:32.507732 6821 webserver.cc:460] Webserver started at http://127.0.98.3:39811/ using document root <none> and password file <none>
I20260926 13:14:32.507968 6821 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.508038 6821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.508195 6821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:32.509344 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "73d7b08afc7b4173996a74341328a3ce"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "12e07b621a936a60ab8568edf438a0c3"
server_key_iv: "32f5cf9883ef71a015970450804a74c7"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.509764 6821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73d7b08afc7b4173996a74341328a3ce"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "12e07b621a936a60ab8568edf438a0c3"
server_key_iv: "32f5cf9883ef71a015970450804a74c7"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.511315 6821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:14:32.512039 6836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.512207 6821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:32.512302 6821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "73d7b08afc7b4173996a74341328a3ce"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "12e07b621a936a60ab8568edf438a0c3"
server_key_iv: "32f5cf9883ef71a015970450804a74c7"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.512385 6821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.522718 6821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:32.525115 6821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:32.525282 6821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:32.525535 6821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:32.525839 6821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:32.525894 6821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.525945 6821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:32.525993 6821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.531998 6821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35047
I20260926 13:14:32.532037 6949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35047 every 8 connection(s)
I20260926 13:14:32.532363 6821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:32.536221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6821
I20260926 13:14:32.536353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:14:32.536556 392 external_mini_cluster.cc:1428] Setting key 38ca514830b9404a81af42c7de128ae9
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:32.542444 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.533102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44211 (local address 127.0.98.62:45117)
0926 13:14:32.533190 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.533193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.533389 (+ 196us) server_negotiation.cc:407] Connection header received
0926 13:14:32.533940 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.533941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.533954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.534030 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.535094 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.535595 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.536709 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.536944 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.537018 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.539420 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.539421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.539422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.539430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.540927 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.541405 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.541406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.541406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.541467 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.541727 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.541728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.541728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.541844 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.541916 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.542273 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.542355 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:14:32.542634 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.533060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:44211)
0926 13:14:32.533245 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.533291 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.533821 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.534105 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.534108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.534755 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.534980 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.534985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.535704 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.535705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.536639 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.536642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.537054 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.537055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.537215 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.537836 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.537844 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.539323 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.541017 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.541024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.541027 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.541331 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.541535 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.541535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.541536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.541654 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.541918 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.541925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.542085 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.542223 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.542385 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260926 13:14:32.543210 6950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45117
I20260926 13:14:32.543306 6950 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:32.543467 6950 heartbeater.cc:511] Master 127.0.98.62:45117 requested a full tablet report, sending...
I20260926 13:14:32.543820 6491 ts_manager.cc:194] Registered new tserver with Master: 73d7b08afc7b4173996a74341328a3ce (127.0.98.3:35047)
I20260926 13:14:32.544154 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44211
I20260926 13:14:32.548630 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 13:14:32.575507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39897
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45117,127.0.98.61:39897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:32.665643 6955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:32.665906 6955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:32.665934 6955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:32.667336 6955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:32.667393 6955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:32.667410 6955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:32.667424 6955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:32.667438 6955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:32.669368 6955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45117,127.0.98.61:39897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
W20260926 13:14:32.669500 6955 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:45117,127.0.98.61:39897'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 13:14:32.669721 6955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:32.670001 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:14:32.676214 6955 server_base.cc:1034] running on GCE node
W20260926 13:14:32.676214 6963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.676214 6961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.676462 6960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.676749 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:32.677088 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:32.678238 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790428472678224 us; error 28 us; skew 500 ppm
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.681120 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 13:14:32.681695 6955 webserver.cc:460] Webserver started at http://127.0.98.61:39911/ using document root <none> and password file <none>
I20260926 13:14:32.681974 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.682053 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.682257 6955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:32.683487 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/instance:
uuid: "20c3714dbe9e4b50b61b94d47ca0f762"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "2296a8d647fad7c8e55e34e27ea2d12d"
server_key_iv: "9b6e2dc1e849593be01758a11fad0660"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.683944 6955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal/instance:
uuid: "20c3714dbe9e4b50b61b94d47ca0f762"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "2296a8d647fad7c8e55e34e27ea2d12d"
server_key_iv: "9b6e2dc1e849593be01758a11fad0660"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.685477 6955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:32.686453 6970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.686679 6955 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:32.686800 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
uuid: "20c3714dbe9e4b50b61b94d47ca0f762"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "2296a8d647fad7c8e55e34e27ea2d12d"
server_key_iv: "9b6e2dc1e849593be01758a11fad0660"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.686882 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.700268 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:32.702775 6955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:32.702941 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:32.706952 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39897
I20260926 13:14:32.706986 7022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39897 every 8 connection(s)
I20260926 13:14:32.707374 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:14:32.708448 7023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:32.709988 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:14:32.711122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 6955
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:14:32.720566 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.710888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.0.98.62:45117)
0926 13:14:32.710946 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.710948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.711248 (+ 300us) server_negotiation.cc:407] Connection header received
0926 13:14:32.712074 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.712076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.712091 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.712161 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.712759 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.713309 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.714377 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.714579 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.714668 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.717249 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.717250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.717250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.717256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.719017 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.719445 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.719446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.719447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.719506 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.719738 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.719739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.719739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.719968 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.720039 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.720384 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.720477 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:14:32.720768 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.710876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54722)
0926 13:14:32.711045 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.711117 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.711948 (+ 831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.712180 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.712185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.712470 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.712647 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.712652 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.713437 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.713438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.714268 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.714271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.714676 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.714677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.714815 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.715446 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.715456 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.717160 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.719109 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.719116 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.719119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.719366 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.719577 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.719578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.719578 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.719674 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.720057 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.720064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.720234 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.720353 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.720513 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260926 13:14:32.721359 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.711757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.0.98.61:39897)
0926 13:14:32.711894 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.711899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.711943 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:14:32.712066 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.712068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.712095 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.712196 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.712804 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.713622 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.714438 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.714766 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.714852 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.717950 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.717958 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.717958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.717965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.719879 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.720359 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.720362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.720362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.720422 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.720750 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.720751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.720751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.720933 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.721092 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.721172 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.721249 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260926 13:14:32.721449 7023 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:14:32.727473 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.721693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.0.1:40276)
0926 13:14:32.721734 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.721744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.721776 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.721954 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.721955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.722084 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.722127 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.722128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.722753 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.722754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.723228 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.723229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.723581 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.723582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.723650 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.724097 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.724102 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.725250 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.726616 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.726616 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.726617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.726777 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.726943 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.726943 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.726944 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.726991 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.727188 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.727189 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.727234 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.727302 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.727369 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:14:32.727478 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.721752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.98.61:39897)
0926 13:14:32.721843 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.721845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.721852 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:32.721868 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.721869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.721879 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.721931 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.722239 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.722692 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.723292 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.723500 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.723553 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.725325 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.725326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.725326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.725332 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.726547 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.726843 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.726843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.726844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.726889 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.727036 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.727037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.727037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.727117 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.727159 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.727331 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.727410 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 13:14:32.728968 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Bootstrap starting.
I20260926 13:14:32.729794 7023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:32.730074 7023 log.cc:824] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:32.730962 7023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: No bootstrap required, opened a new log
I20260926 13:14:32.732560 7023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } }
I20260926 13:14:32.732733 7023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:32.732785 7023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20c3714dbe9e4b50b61b94d47ca0f762, State: Initialized, Role: FOLLOWER
I20260926 13:14:32.732952 7023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } }
I20260926 13:14:32.733253 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } } }
I20260926 13:14:32.733337 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:14:32.733355 7023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 13:14:32.735499 7040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:32.735582 7040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:32.737110 6955 master_runner.cc:428] Detected that this master 20c3714dbe9e4b50b61b94d47ca0f762 is joining an existing cluster
I20260926 13:14:32.737167 6955 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:39897
I20260926 13:14:32.745528 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.738080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54732)
0926 13:14:32.738253 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.738264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.738350 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.738536 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.738536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.738695 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.738809 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.738812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.739748 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.739749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.740666 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.740668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.741258 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.741259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.741363 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.741950 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.741955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.742313 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.744265 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.744266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.744267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.744508 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.744797 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.744798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.744799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.744865 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.745172 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.745173 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.745222 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.745301 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.745407 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":118,"threads_started":1}
I20260926 13:14:32.745517 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.738198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.0.98.62:45117)
0926 13:14:32.738250 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.738253 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.738335 (+ 82us) server_negotiation.cc:407] Connection header received
0926 13:14:32.738432 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.738433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.738444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.738474 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.738877 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.739659 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.740757 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.741106 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.741177 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.742392 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.742393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.742393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.742400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.744175 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.744613 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.744615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.744616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.744680 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.744951 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.744952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.744952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.745094 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.745167 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.745343 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.745421 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:32.746230 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.738396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.0.98.61:39897)
0926 13:14:32.738620 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.738622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.738749 (+ 127us) server_negotiation.cc:407] Connection header received
0926 13:14:32.738805 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.738806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.738814 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.738870 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.740169 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.740830 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.741580 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.741881 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.741945 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.743285 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.743286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.743286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.743295 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.744903 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.745221 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.745222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.745222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.745279 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.745472 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.745472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.745472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.745585 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.745627 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.746038 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.746114 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:32.746797 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.738337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.0.1:40292)
0926 13:14:32.738673 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.738682 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.738753 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.739765 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.739766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.740020 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.740088 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.740090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.740910 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.740911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.741496 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.741498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.742386 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.742387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.742435 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.742835 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.742839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.743184 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.744975 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.744975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.744975 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.745119 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.745350 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.745351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.745351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.745405 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.745885 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.745886 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.745924 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.746446 (+ 522us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.746537 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":55,"threads_started":1}
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:14:32.755818 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.747940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.0.1:40306)
0926 13:14:32.748144 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.748176 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.748233 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.748374 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.748376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.748618 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.748666 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.748667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.749574 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.749575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.750230 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.750232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.750660 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.750660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.750750 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.751253 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.751259 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.752831 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.754625 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.754633 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.754635 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.754844 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.755085 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.755086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.755086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.755147 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.755416 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.755424 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.755477 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.755580 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.755703 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260926 13:14:32.755813 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.748012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.0.98.61:39897)
0926 13:14:32.748050 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.748052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.748235 (+ 183us) server_negotiation.cc:407] Connection header received
0926 13:14:32.748295 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.748296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.748305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.748371 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.748734 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.749488 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.750354 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.750571 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.750659 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.752926 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.752927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.752927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.752936 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.754523 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.754951 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.754952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.754953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.755010 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.755218 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.755219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.755220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.755339 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.755411 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.755619 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.755698 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:14:32.756583 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:39897: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 39897
}
attrs {
promote: true
}
}
dest_uuid: "25e06fc9a29a471182200dfd21afc633"
cas_config_opid_index: -1
I20260926 13:14:32.756933 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } }
I20260926 13:14:32.757424 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 20c3714dbe9e4b50b61b94d47ca0f762 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } }
I20260926 13:14:32.757802 6997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:32.758587 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:39897
I20260926 13:14:32.758785 6997 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Refusing update from remote peer 25e06fc9a29a471182200dfd21afc633: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 13:14:32.758846 6955 master.cc:537] Master@127.0.98.61:39897 shutting down...
I20260926 13:14:32.758845 7049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } }
I20260926 13:14:32.758908 7049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:32.759169 7050 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:14:32.759933 6478 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 13:14:32.761788 6955 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:32.761910 6955 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:32.761957 6955 tablet_replica.cc:330] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: stopping tablet replica
W20260926 13:14:32.762570 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 -> Peer 20c3714dbe9e4b50b61b94d47ca0f762 (127.0.98.61:39897): Couldn't send request to peer 20c3714dbe9e4b50b61b94d47ca0f762. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:14:32.774890 6955 master.cc:552] Master@127.0.98.61:39897 shutdown complete.
I20260926 13:14:32.774974 6955 master_runner.cc:305] Clearing existing system tablet
I20260926 13:14:32.775111 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.775178 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.775970 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:32.776552 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.776719 6955 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.776788 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:14:32.776816 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
uuid: "20c3714dbe9e4b50b61b94d47ca0f762"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "2296a8d647fad7c8e55e34e27ea2d12d"
server_key_iv: "9b6e2dc1e849593be01758a11fad0660"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.778196 6955 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:32.779965 6955 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:14:32.780051 6955 log.cc:1196] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 13:14:32.780254 6955 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Deleting consensus metadata
I20260926 13:14:32.780380 6955 master_runner.cc:315] Copying system tablet from 127.0.98.62:45117
I20260926 13:14:32.781052 6955 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45117
I20260926 13:14:32.787895 7060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.781429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54734)
0926 13:14:32.781603 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:32.781617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:32.781696 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:32.781858 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:32.781859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:32.782068 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:32.782142 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.782145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.782950 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.782951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.783583 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:32.783586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.783987 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.783988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:32.784058 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.784573 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:32.784578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:32.784965 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:32.786767 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.786768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.786768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.786959 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.787171 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:32.787172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:32.787173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:32.787232 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:32.787507 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:32.787508 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:32.787555 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 13:14:32.787646 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 13:14:32.787758 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260926 13:14:32.787918 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.781492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.0.98.62:45117)
0926 13:14:32.781566 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.781568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.781677 (+ 109us) server_negotiation.cc:407] Connection header received
0926 13:14:32.781762 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.781763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.781773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.781830 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.782343 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.782847 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.783685 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.783886 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.783977 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.785061 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.785063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.785063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.785071 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.786660 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.787037 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.787038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.787038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.787102 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.787321 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.787322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.787322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.787439 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.787515 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.787673 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.787756 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:14:32.788244 6511 tablet_copy_service.cc:140] P 25e06fc9a29a471182200dfd21afc633: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 20c3714dbe9e4b50b61b94d47ca0f762 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54734)
I20260926 13:14:32.788350 6511 tablet_copy_service.cc:161] P 25e06fc9a29a471182200dfd21afc633: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 20c3714dbe9e4b50b61b94d47ca0f762 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54734: session id = 20c3714dbe9e4b50b61b94d47ca0f762-00000000000000000000000000000000
I20260926 13:14:32.789007 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:14:32.789713 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:14:32.791114 6955 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: tablet copy: Starting download of 0 data blocks...
I20260926 13:14:32.791281 6955 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: tablet copy: Starting download of 1 WAL segments...
I20260926 13:14:32.792409 6955 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:14:32.793473 6511 tablet_copy_service.cc:342] P 25e06fc9a29a471182200dfd21afc633: Request end of tablet copy session 20c3714dbe9e4b50b61b94d47ca0f762-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54734
I20260926 13:14:32.793541 6511 tablet_copy_service.cc:433] P 25e06fc9a29a471182200dfd21afc633: ending tablet copy session 20c3714dbe9e4b50b61b94d47ca0f762-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 20c3714dbe9e4b50b61b94d47ca0f762
I20260926 13:14:32.796738 6955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:32.798049 7062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.798329 7064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.798326 7061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.798328 6955 server_base.cc:1034] running on GCE node
I20260926 13:14:32.798525 6955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:32.798662 6955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:32.799820 6955 hybrid_clock.cc:648] HybridClock initialized: now 1790428472799799 us; error 49 us; skew 500 ppm
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.801407 6955 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 13:14:32.801715 6955 webserver.cc:460] Webserver started at http://127.0.98.61:34831/ using document root <none> and password file <none>
I20260926 13:14:32.801862 6955 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.801913 6955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.802640 6955 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:32.803176 7071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.803299 6955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:32.803347 6955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
uuid: "20c3714dbe9e4b50b61b94d47ca0f762"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "2296a8d647fad7c8e55e34e27ea2d12d"
server_key_iv: "9b6e2dc1e849593be01758a11fad0660"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.803406 6955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.815933 6955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:32.817718 6955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:32.820799 6955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39897
I20260926 13:14:32.820806 7123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39897 every 8 connection(s)
I20260926 13:14:32.821138 6955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:14:32.822027 7124 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:14:32.822600 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Bootstrap starting.
I20260926 13:14:32.826050 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:14:32.826303 7124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: Bootstrap complete.
I20260926 13:14:32.826511 7124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } }
I20260926 13:14:32.826604 7124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20c3714dbe9e4b50b61b94d47ca0f762, State: Initialized, Role: LEARNER
I20260926 13:14:32.826644 7124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } }
I20260926 13:14:32.826831 7124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:32.828289 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } }
I20260926 13:14:32.828399 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:14:32.829197 7139 catalog_manager.cc:1259] Loaded cluster ID: 2d35d30aae284e0eb9f9a552b2cafde8
I20260926 13:14:32.829260 7139 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: loading cluster ID for follower catalog manager: success
I20260926 13:14:32.830001 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:32.822936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.0.98.61:39897)
0926 13:14:32.823071 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:32.823073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:32.823083 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:32.823116 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:32.823117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:32.823126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:32.823223 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:32.823771 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.824558 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.825263 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:32.825490 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:32.825570 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:32.827033 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:32.827034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:32.827035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:32.827042 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:32.828839 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.829201 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.829202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.829203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.829301 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:32.829490 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:32.829491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:32.829491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:32.829608 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:32.829714 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:32.829822 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 13:14:32.829901 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":90,"threads_started":1}
I20260926 13:14:32.830358 7139 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: acquiring CA information for follower catalog manager: success
I20260926 13:14:32.830670 7139 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 13:14:32.861873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43189
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:32.956696 7142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:32.956914 7142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:32.956936 7142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:32.958133 7142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:14:32.958200 7142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:32.958258 7142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:32.958287 7142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:14:32.958304 7142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:14:32.959846 7142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:32.960150 7142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:32.960455 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:32.965978 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:32.965981 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.966012 7142 server_base.cc:1034] running on GCE node
W20260926 13:14:32.966178 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:32.966459 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:32.966845 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:32.968027 7142 hybrid_clock.cc:648] HybridClock initialized: now 1790428472968003 us; error 41 us; skew 500 ppm
Sep 26 13:14:32 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:32.970808 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 13:14:32.971274 7142 webserver.cc:460] Webserver started at http://127.0.98.60:39743/ using document root <none> and password file <none>
I20260926 13:14:32.971521 7142 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:32.971591 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:32.971740 7142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:14:32.972863 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/instance:
uuid: "5669b0b0812e472f95acaa3fb78f6561"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "5a0af4a3c41bed471cb74b340c15635a"
server_key_iv: "5cb3201c6f04e4e683c538e5ee0c403d"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.973243 7142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal/instance:
uuid: "5669b0b0812e472f95acaa3fb78f6561"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "5a0af4a3c41bed471cb74b340c15635a"
server_key_iv: "5cb3201c6f04e4e683c538e5ee0c403d"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.974756 7142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:14:32.975703 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:32.975921 7142 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:32.976032 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
uuid: "5669b0b0812e472f95acaa3fb78f6561"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "5a0af4a3c41bed471cb74b340c15635a"
server_key_iv: "5cb3201c6f04e4e683c538e5ee0c403d"
server_key_version: "encryptionkey@0"
I20260926 13:14:32.976114 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:32.998518 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:33.001227 7142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:33.001418 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:33.006148 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43189
I20260926 13:14:33.006212 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43189 every 8 connection(s)
I20260926 13:14:33.006770 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:14:33.007498 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:33.007822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 7142
I20260926 13:14:33.009783 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:14:33.019546 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.010726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.98.62:45117)
0926 13:14:33.010773 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.010775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.010784 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:14:33.010857 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.010858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.010868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.010939 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.011640 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.012289 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.013197 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.013420 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.013517 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.016080 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.016082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.016082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.016090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.017573 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.018498 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.018499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.018499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.018549 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.018858 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.018858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.018859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.018958 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.019050 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.019351 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.019442 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:14:33.020043 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.010555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54748)
0926 13:14:33.010703 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.010743 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.010809 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.010996 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.010999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.011436 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.011538 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.011543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.012410 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.012411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.013094 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.013096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.013530 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.013531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.013585 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.013955 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.013977 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.015989 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.017658 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.017665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.017667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.018412 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.018620 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.018621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.018621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.018786 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.019055 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.019059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.019192 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.019333 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.019510 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260926 13:14:33.020203 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.008647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.0.98.60:43189)
0926 13:14:33.008849 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.008859 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.008898 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:14:33.009475 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.009485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.009554 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.009711 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.010550 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.011908 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.012729 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.013121 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.013244 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.016305 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.016312 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.016312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.016318 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.018949 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.019287 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.019290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.019290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.019356 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.019593 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.019593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.019594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.019762 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.019902 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.019954 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.020029 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
I20260926 13:14:33.020800 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 13:14:33.027788 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.021071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.0.1:40334)
0926 13:14:33.021122 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.021132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.021172 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.021360 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.021362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.021501 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.021546 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.021547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.022297 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.022298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.022804 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.022805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.023221 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.023222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.023297 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.023732 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.023737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.024849 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.026721 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.026726 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.026726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.026922 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.027138 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.027139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.027139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.027234 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.027536 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.027537 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.027571 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.027621 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.027690 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:14:33.027871 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.021144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40334 (local address 127.0.98.61:39897)
0926 13:14:33.021194 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.021197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.021227 (+ 30us) server_negotiation.cc:407] Connection header received
0926 13:14:33.021258 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.021259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.021273 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.021357 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.021628 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.022136 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.022908 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.023131 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.023222 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.024931 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.024932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.024932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.024942 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.026590 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.026984 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.026985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.026985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.027040 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.027328 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.027329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.027329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.027454 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.027549 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.027650 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.027745 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:33.028261 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43189 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428472, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:14:33.034279 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.028551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43189 (local address 127.0.0.1:44866)
0926 13:14:33.028598 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.028608 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.028646 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.028775 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.028776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.028879 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.028907 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.028908 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.029472 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.029473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.030014 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.030017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.030492 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.030493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.030545 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.030908 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.030913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.032159 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.033455 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.033455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.033456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.033608 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.033772 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.033773 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.033773 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.033821 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.033991 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.033992 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.034031 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.034080 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.034153 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:14:33.034346 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.028621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.0.98.60:43189)
0926 13:14:33.028656 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.028657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.028688 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:14:33.028713 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.028714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.028723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.028772 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.028949 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.029410 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.030107 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.030415 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.030479 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.032238 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.032239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.032240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.032247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.033389 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.033671 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.033672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.033672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.033717 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.033862 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.033862 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.033863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.033943 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.033996 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.034094 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.034266 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":16}
I20260926 13:14:33.036057 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Bootstrap starting.
I20260926 13:14:33.036623 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:33.036844 7210 log.cc:824] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:33.037513 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: No bootstrap required, opened a new log
I20260926 13:14:33.038924 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43189 } }
I20260926 13:14:33.039054 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:33.039077 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5669b0b0812e472f95acaa3fb78f6561, State: Initialized, Role: FOLLOWER
I20260926 13:14:33.039198 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43189 } }
I20260926 13:14:33.039462 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43189 } } }
I20260926 13:14:33.039541 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:33.039544 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:14:33.041709 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:14:33.041769 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:14:33.045887 7142 master_runner.cc:428] Detected that this master 5669b0b0812e472f95acaa3fb78f6561 is joining an existing cluster
I20260926 13:14:33.045941 7142 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43189
I20260926 13:14:33.056289 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.047230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.0.98.62:45117)
0926 13:14:33.047295 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.047298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.047599 (+ 301us) server_negotiation.cc:407] Connection header received
0926 13:14:33.047709 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.047710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.047722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.047758 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.048403 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.049201 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.050518 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.050821 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.050902 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.052165 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.052166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.052167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.052173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.054108 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.054650 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.054651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.054652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.054711 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.054948 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.054949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.054949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.055528 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.055579 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.056059 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.056155 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:33.057139 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.047164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54754)
0926 13:14:33.047532 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.047543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.047627 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.047839 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.047841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.048076 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.048168 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.048170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.049351 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.049351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.050404 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.050406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.050913 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.050914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.050998 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.051601 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.051607 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.052064 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.054252 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.054253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.054253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.054555 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.054787 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.054788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.054788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.054871 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.055633 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.055634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.055680 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.056422 (+ 742us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.057002 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":77,"threads_started":1}
I20260926 13:14:33.058521 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.047611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.98.60:43189)
0926 13:14:33.047664 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.047667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.048251 (+ 584us) server_negotiation.cc:407] Connection header received
0926 13:14:33.048274 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.048275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.048285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.048345 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.048948 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.049661 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.051287 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.051585 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.051629 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.054794 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.054794 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.054795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.054800 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.056917 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.057516 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.057517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.057518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.057568 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.057822 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.057823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.057824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.057954 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.058049 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.058365 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.058438 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:14:33.058943 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.047599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43189 (local address 127.0.0.1:44878)
0926 13:14:33.048078 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.048088 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.048136 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.048686 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.048687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.048842 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.048888 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.048889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.050102 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.050103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.051197 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.051199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.052816 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.052817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.052894 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.054268 (+ 1374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.054272 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.054713 (+ 441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.057198 (+ 2485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.057199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.057199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.057413 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.057630 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.057630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.057631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.057677 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.058130 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.058131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.058260 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.058624 (+ 364us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.058843 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":77,"threads_started":1}
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428471, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:14:33.071759 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6553
I20260926 13:14:33.080950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33261
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45117
--webserver_port=41895
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:14:33.081017 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.047429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.0.98.61:39897)
0926 13:14:33.047472 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.047474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.047776 (+ 302us) server_negotiation.cc:407] Connection header received
0926 13:14:33.047881 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.047882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.047893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.047923 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.048987 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.049917 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.053084 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.053408 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.053449 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.077817 (+ 24368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.077820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.077820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.077829 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.079632 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.080132 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.080134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.080134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.080191 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.080404 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.080405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.080405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.080555 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.080597 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.080828 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.080914 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:14:33.081944 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.047367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.0.1:40344)
0926 13:14:33.047722 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.047732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.047808 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.047983 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.047983 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.048546 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.048598 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.048599 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.051606 (+ 3007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.051607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.052710 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.052712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.053524 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.053525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.053570 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.076386 (+ 22816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.076392 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.077690 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.079737 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.079738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.079739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.080041 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.080266 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.080267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.080267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.080322 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.080642 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.080643 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.080711 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.081644 (+ 933us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.081781 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":71,"threads_started":1}
I20260926 13:14:33.083264 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.060194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.0.98.60:43189)
0926 13:14:33.060231 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.060233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.060274 (+ 41us) server_negotiation.cc:407] Connection header received
0926 13:14:33.060350 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.060350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.060360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.060407 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.060725 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.061430 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.062602 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.062975 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.063067 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.065408 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.065410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.065410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.065417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.077236 (+ 11819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.082394 (+ 5158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.082395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.082396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.082457 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.082688 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.082689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.082689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.082816 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.082928 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.083104 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.083181 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:14:33.083290 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.060159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43189 (local address 127.0.0.1:44884)
0926 13:14:33.060233 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.060244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.060290 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.060433 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.060434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.060584 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.060648 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.060650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.061552 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.061553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.062512 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.062527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.063074 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.063074 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.063160 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.063677 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.063682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.065303 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.079957 (+ 14654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.079958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.079959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.080225 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.082542 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.082544 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.082544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.082618 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.082922 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.082923 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.082977 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.083080 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.083190 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 13:14:33.083976 6491 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43189: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43189
}
attrs {
promote: true
}
}
dest_uuid: "25e06fc9a29a471182200dfd21afc633"
cas_config_opid_index: 5
I20260926 13:14:33.084254 6491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:33.084589 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5669b0b0812e472f95acaa3fb78f6561 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.085095 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:33.085563 7050 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43189
I20260926 13:14:33.085631 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.085688 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:33.085954 7142 master.cc:537] Master@127.0.98.60:43189 shutting down...
I20260926 13:14:33.086089 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 1 FOLLOWER]: Refusing update from remote peer 25e06fc9a29a471182200dfd21afc633: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:14:33.087067 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 13:14:33.087698 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 -> Peer 5669b0b0812e472f95acaa3fb78f6561 (127.0.98.60:43189): Couldn't send request to peer 5669b0b0812e472f95acaa3fb78f6561. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:14:33.090219 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.084712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.0.1:40354)
0926 13:14:33.086664 (+ 1952us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.086676 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.086726 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.086955 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.086956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.086958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:33.087004 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.087005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.087967 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.087968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.089138 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.089141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.089887 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.089888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.089968 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.089992 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.090051 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.090105 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1938}
I20260926 13:14:33.090579 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.084798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.0.98.61:39897)
0926 13:14:33.086560 (+ 1762us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.086563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.086779 (+ 216us) server_negotiation.cc:407] Connection header received
0926 13:14:33.086802 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.086803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.086827 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.086896 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:33.087073 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.087772 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.089227 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.089708 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.089767 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.089848 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.090449 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.090497 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736}
I20260926 13:14:33.091023 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 LEARNER]: Refusing update from remote peer 25e06fc9a29a471182200dfd21afc633: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 13:14:33.091229 7049 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 13:14:33.091486 7142 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:33.091575 7142 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:33.091619 7142 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: stopping tablet replica
I20260926 13:14:33.091797 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5669b0b0812e472f95acaa3fb78f6561 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.092824 7128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.092931 7128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:14:33.092823 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25e06fc9a29a471182200dfd21afc633. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: true } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.093025 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:14:33.093554 7049 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: attempting to promote NON_VOTER 20c3714dbe9e4b50b61b94d47ca0f762 to VOTER
I20260926 13:14:33.093765 7049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:33.094313 7098 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 LEARNER]: Refusing update from remote peer 25e06fc9a29a471182200dfd21afc633: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 13:14:33.094586 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 -> Peer 5669b0b0812e472f95acaa3fb78f6561 (127.0.98.60:43189): Couldn't send request to peer 5669b0b0812e472f95acaa3fb78f6561. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 13:14:33.095021 7238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 13:14:33.095814 7050 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 20c3714dbe9e4b50b61b94d47ca0f762 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.095947 7098 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 20c3714dbe9e4b50b61b94d47ca0f762 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.096809 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.096908 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:33.096899 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.096971 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:14:33.096971 7050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "25e06fc9a29a471182200dfd21afc633" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.097024 7050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633 [sys.catalog]: This master's current role is: LEADER
I20260926 13:14:33.107052 7142 master.cc:552] Master@127.0.98.60:43189 shutdown complete.
I20260926 13:14:33.107107 7142 master_runner.cc:305] Clearing existing system tablet
I20260926 13:14:33.107239 7142 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:33.107291 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:33.107990 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:33.108515 7242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.108660 7142 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:33.108705 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.108726 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
uuid: "5669b0b0812e472f95acaa3fb78f6561"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "5a0af4a3c41bed471cb74b340c15635a"
server_key_iv: "5cb3201c6f04e4e683c538e5ee0c403d"
server_key_version: "encryptionkey@0"
I20260926 13:14:33.109513 7142 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:33.111192 7142 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 13:14:33.111251 7142 log.cc:1196] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 13:14:33.111446 7142 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Deleting consensus metadata
I20260926 13:14:33.111553 7142 master_runner.cc:315] Copying system tablet from 127.0.98.62:45117
I20260926 13:14:33.112135 7142 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:45117
I20260926 13:14:33.119603 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.112499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.0.98.62:45117)
0926 13:14:33.112546 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.112549 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.112697 (+ 148us) server_negotiation.cc:407] Connection header received
0926 13:14:33.112795 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.112796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.112806 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.112846 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.113262 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.114024 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.115179 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.115458 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.115498 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.116699 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.116701 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.116701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.116708 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.118382 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.118781 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.118782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.118782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.118850 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.119067 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.119068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.119068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.119201 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.119268 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.119429 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.119513 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:33.119647 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.112510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54762)
0926 13:14:33.112619 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.112631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.112722 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.112910 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.112911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.113109 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.113176 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.113178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.114130 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.114131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.115081 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.115083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.115572 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.115573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.115636 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.116207 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.116212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.116597 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.118462 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.118463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.118470 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.118693 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.118925 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.118926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.118926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.118994 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.119270 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.119271 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.119321 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.119394 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.119511 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":73,"threads_started":1}
I20260926 13:14:33.120199 6511 tablet_copy_service.cc:140] P 25e06fc9a29a471182200dfd21afc633: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5669b0b0812e472f95acaa3fb78f6561 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54762)
I20260926 13:14:33.120253 6511 tablet_copy_service.cc:161] P 25e06fc9a29a471182200dfd21afc633: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5669b0b0812e472f95acaa3fb78f6561 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54762: session id = 5669b0b0812e472f95acaa3fb78f6561-00000000000000000000000000000000
I20260926 13:14:33.120806 6511 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 13:14:33.121328 7142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 13:14:33.122629 7142 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: tablet copy: Starting download of 0 data blocks...
I20260926 13:14:33.122778 7142 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: tablet copy: Starting download of 1 WAL segments...
I20260926 13:14:33.123586 7142 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 13:14:33.124413 6511 tablet_copy_service.cc:342] P 25e06fc9a29a471182200dfd21afc633: Request end of tablet copy session 5669b0b0812e472f95acaa3fb78f6561-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54762
I20260926 13:14:33.124466 6511 tablet_copy_service.cc:433] P 25e06fc9a29a471182200dfd21afc633: ending tablet copy session 5669b0b0812e472f95acaa3fb78f6561-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5669b0b0812e472f95acaa3fb78f6561
I20260926 13:14:33.129554 7142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:33.130749 7252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:33.130789 7249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:33.131073 7250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:33.131373 7142 server_base.cc:1034] running on GCE node
I20260926 13:14:33.131470 7142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:33.131585 7142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:33.132705 7142 hybrid_clock.cc:648] HybridClock initialized: now 1790428473132685 us; error 34 us; skew 500 ppm
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:33.134279 7142 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 13:14:33.134523 7142 webserver.cc:460] Webserver started at http://127.0.98.60:44251/ using document root <none> and password file <none>
I20260926 13:14:33.134629 7142 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:33.134654 7142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:33.135331 7142 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:14:33.135880 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.135989 7142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.136029 7142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
uuid: "5669b0b0812e472f95acaa3fb78f6561"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "5a0af4a3c41bed471cb74b340c15635a"
server_key_iv: "5cb3201c6f04e4e683c538e5ee0c403d"
server_key_version: "encryptionkey@0"
I20260926 13:14:33.136087 7142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:33.144742 7142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:33.146700 7142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:33.149638 7142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43189
I20260926 13:14:33.150013 7142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:14:33.152959 7311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43189 every 8 connection(s)
I20260926 13:14:33.153666 7312 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:14:33.154227 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Bootstrap starting.
I20260926 13:14:33.157089 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:14:33.157333 7312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: Bootstrap complete.
I20260926 13:14:33.157544 7312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:33.157634 7312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5669b0b0812e472f95acaa3fb78f6561, State: Initialized, Role: LEARNER
I20260926 13:14:33.157670 7312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:33.157882 7312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:14:33.157882 7315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } } }
I20260926 13:14:33.158118 7315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561 [sys.catalog]: This master's current role is: LEARNER
I20260926 13:14:33.159483 7326 catalog_manager.cc:1259] Loaded cluster ID: 2d35d30aae284e0eb9f9a552b2cafde8
I20260926 13:14:33.159571 7326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: loading cluster ID for follower catalog manager: success
I20260926 13:14:33.160858 7326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: acquiring CA information for follower catalog manager: success
I20260926 13:14:33.161155 7326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5669b0b0812e472f95acaa3fb78f6561: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 13:14:33.188815 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:33.189066 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:33.189108 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:33.190347 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:33.190418 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:33.190466 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:14:33.191977 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41895
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:33.192255 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:33.192503 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:33.198657 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:33.198683 7235 server_base.cc:1034] running on GCE node
W20260926 13:14:33.198665 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:33.198920 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:33.199111 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:33.199419 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:33.200620 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790428473200605 us; error 31 us; skew 500 ppm
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:33.203444 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:14:33.204010 7235 webserver.cc:460] Webserver started at http://127.0.98.1:41895/ using document root <none> and password file <none>
I20260926 13:14:33.204290 7235 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:33.204349 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:33.205796 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:14:33.206496 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.206744 7235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:14:33.206854 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "1affc48a249337901d3a25c619f4ce64"
server_key_iv: "fd64cf88bf9556b90227f7e6a17a0626"
server_key_version: "encryptionkey@0"
I20260926 13:14:33.207253 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:33.220027 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:33.222532 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:33.222700 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:33.222945 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:33.223258 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:33.223311 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.223357 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:33.223397 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.230348 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33261
I20260926 13:14:33.230397 7454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33261 every 8 connection(s)
I20260926 13:14:33.230764 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:33.240058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 7235
I20260926 13:14:33.240919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6687
I20260926 13:14:33.242758 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.231488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43307 (local address 127.0.98.61:39897)
0926 13:14:33.231556 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.231559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.231761 (+ 202us) server_negotiation.cc:407] Connection header received
0926 13:14:33.232845 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.232847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.232857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.232942 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.233915 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.234718 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.235961 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.236234 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.236319 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.239163 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.239164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.239164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.239173 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.241025 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.241514 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.241515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.241516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.241573 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.241898 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.241899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.241900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.242078 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.242123 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.242566 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.242649 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:14:33.243337 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.231453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.98.1:43307)
0926 13:14:33.231612 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.231656 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.232741 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.233044 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.233056 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.233570 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.233795 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.233801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.234821 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.234822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.235847 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.235849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.236325 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.236326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.236459 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.237283 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.237288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.239043 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.241124 (+ 2081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.241132 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.241135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.241452 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.241648 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.241649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.241649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.241805 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.242196 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.242202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.242384 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.242868 (+ 484us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.243067 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":10,"thread_start_us":94,"threads_started":1}
I20260926 13:14:33.243983 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.232018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41335 (local address 127.0.98.60:43189)
0926 13:14:33.232131 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.232133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.233185 (+ 1052us) server_negotiation.cc:407] Connection header received
0926 13:14:33.233209 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.233210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.233220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.233294 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.233908 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.234657 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.235928 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.236269 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.236330 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.239606 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.239607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.239607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.239616 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.242487 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.242917 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.242918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.242918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.242976 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.243165 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.243166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.243167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.243351 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.243587 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.243783 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.243875 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260926 13:14:33.244024 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.231929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43189 (local address 127.0.98.1:41335)
0926 13:14:33.233075 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.233085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.233127 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.233300 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.233300 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.233572 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.233795 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.233800 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.234764 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.234765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.235807 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.235809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.236386 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.236387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.236459 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.237193 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.237204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.239509 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.242579 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.242580 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.242581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.242838 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.243043 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.243043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.243044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.243104 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.243614 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.243615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.243663 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.243736 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.243864 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":54,"thread_start_us":61,"threads_started":1}
I20260926 13:14:33.244475 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43189
I20260926 13:14:33.244587 7455 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.244746 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.231826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41075 (local address 127.0.98.62:45117)
0926 13:14:33.232054 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.232057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.232972 (+ 915us) server_negotiation.cc:407] Connection header received
0926 13:14:33.233044 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.233045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.233059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.233123 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.233914 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.234719 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.235964 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.236268 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.236329 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.240113 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.240115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.240115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.240124 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.242540 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.243266 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.243268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.243268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.243328 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.243813 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.243814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.243815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.243947 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.243990 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.244566 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.244653 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260926 13:14:33.244800 7455 heartbeater.cc:511] Master 127.0.98.60:43189 requested a full tablet report, sending...
I20260926 13:14:33.245373 7276 ts_manager.cc:194] Registered new tserver with Master: bc44d8c907bf4f898a3aa007a9a3b9e5 (127.0.98.1:33261)
I20260926 13:14:33.246052 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.231749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:41075)
0926 13:14:33.232899 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.232908 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.232975 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.233141 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.233143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.233571 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.233795 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.233801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.234821 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.234822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.235850 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.235852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.236447 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.236448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.236500 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.237193 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.237204 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.240023 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.242634 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.242635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.242636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.243176 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.243650 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.243651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.243651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.243714 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.244401 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.244402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.244447 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.244522 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.244705 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":91,"thread_start_us":56,"threads_started":1}
I20260926 13:14:33.246470 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45117
I20260926 13:14:33.246541 7456 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.246618 7456 heartbeater.cc:511] Master 127.0.98.62:45117 requested a full tablet report, sending...
I20260926 13:14:33.246783 7457 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39897
I20260926 13:14:33.246840 7457 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.246914 7457 heartbeater.cc:511] Master 127.0.98.61:39897 requested a full tablet report, sending...
I20260926 13:14:33.246905 6491 ts_manager.cc:194] Re-registered known tserver with Master: bc44d8c907bf4f898a3aa007a9a3b9e5 (127.0.98.1:33261)
I20260926 13:14:33.247291 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41075
I20260926 13:14:33.247951 7088 ts_manager.cc:194] Registered new tserver with Master: bc44d8c907bf4f898a3aa007a9a3b9e5 (127.0.98.1:33261)
I20260926 13:14:33.249366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43917
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45117
--webserver_port=37865
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:14:33.332517 7463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:33.332722 7463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:33.332746 7463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:33.333807 7463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:33.333839 7463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:33.333865 7463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:14:33.335350 7463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37865
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:33.335532 7463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:33.335705 7463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:33.340893 7469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:33.340916 7471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:33.341101 7463 server_base.cc:1034] running on GCE node
W20260926 13:14:33.340916 7468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:33.341398 7463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:33.341653 7463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:33.342813 7463 hybrid_clock.cc:648] HybridClock initialized: now 1790428473342801 us; error 44 us; skew 500 ppm
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:33.345425 7463 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:14:33.345853 7463 webserver.cc:460] Webserver started at http://127.0.98.2:37865/ using document root <none> and password file <none>
I20260926 13:14:33.346086 7463 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:33.346167 7463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:33.347599 7463 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:33.348241 7478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.348495 7463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:33.348618 7463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "b290466c423a449b853ba9387ca51b2c"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "8adcd00dec72d4e46615ab1b0afc5592"
server_key_iv: "36e3513f4ddaa07cc38f8366ddd7c453"
server_key_version: "encryptionkey@0"
I20260926 13:14:33.349012 7463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:33.363008 7463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:33.365259 7463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:33.365406 7463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:33.365650 7463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:33.365972 7463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:33.366029 7463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.366071 7463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:33.366098 7463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.372005 7463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43917
I20260926 13:14:33.372061 7591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43917 every 8 connection(s)
I20260926 13:14:33.372435 7463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:14:33.374433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 7463
I20260926 13:14:33.374594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6821
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:33.382714 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35047
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45117
--webserver_port=39811
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 13:14:33.385768 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.373596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40627 (local address 127.0.98.61:39897)
0926 13:14:33.373643 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.373645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.373785 (+ 140us) server_negotiation.cc:407] Connection header received
0926 13:14:33.374273 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.374275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.374288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.374350 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.375373 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.376163 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.377236 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.377586 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.377666 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.382531 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.382532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.382533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.382542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.384368 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384825 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.384826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.384827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.384891 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.385122 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.385124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.385124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.385283 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.385326 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.385541 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.385637 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:14:33.385959 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.373190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39245 (local address 127.0.98.62:45117)
0926 13:14:33.373294 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.373297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.373534 (+ 237us) server_negotiation.cc:407] Connection header received
0926 13:14:33.374262 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.374263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.374276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.374309 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.375246 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.375811 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.377252 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.377470 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.377564 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.382034 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.382035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.382036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.382044 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.383917 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384476 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.384477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.384478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.384540 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384853 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.384854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.384855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.385017 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.385093 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.385745 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.385845 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 13:14:33.386518 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.373504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.98.2:40627)
0926 13:14:33.373724 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.373731 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.374117 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.374787 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.374788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.375237 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.375292 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.375293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.376257 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.376259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.377141 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.377143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.377664 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.377665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.377735 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.378532 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.378545 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.382401 (+ 3856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.384471 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.384472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.384732 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.384969 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.384971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.385039 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.385375 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.385376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.385422 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.385927 (+ 505us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.386200 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":26,"thread_start_us":53,"threads_started":1}
I20260926 13:14:33.386547 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.373188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:39245)
0926 13:14:33.373392 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.373428 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.374108 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.374427 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.374431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.374855 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.375131 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.375138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.375913 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.375913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.377137 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.377140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.377576 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.377577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.377735 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.379539 (+ 1804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.379547 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.381882 (+ 2335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.384011 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384018 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.384022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.384383 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.384615 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.384616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.384616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.384755 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.385126 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.385133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.385341 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.386081 (+ 740us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.386265 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":68,"thread_start_us":116,"threads_started":1}
I20260926 13:14:33.387032 7594 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39897
I20260926 13:14:33.387141 7594 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.387223 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.373246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33015 (local address 127.0.98.60:43189)
0926 13:14:33.373291 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.373294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.373683 (+ 389us) server_negotiation.cc:407] Connection header received
0926 13:14:33.379002 (+ 5319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.379004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.379020 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.379084 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.379492 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.380241 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.381080 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.381317 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.381410 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.383812 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.383814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.383814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.383823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.385693 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.386315 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.386317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.386317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.386380 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.386610 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.386611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.386612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.386765 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.386837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.387031 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.387118 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:14:33.387243 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.373188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43189 (local address 127.0.98.2:33015)
0926 13:14:33.373622 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.373632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.374109 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.379112 (+ 5003us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.379113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.379302 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.379386 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.379389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.380341 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.380342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.380982 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.380984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.381407 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.381408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.381478 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.381981 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.381986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.383681 (+ 1695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.385905 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.385906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.385907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.386215 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.386465 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.386466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.386466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.386527 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.386875 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.386876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.386924 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.386990 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.387099 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":2,"spinlock_wait_cycles":7424,"thread_start_us":98,"threads_started":1}
I20260926 13:14:33.387387 7594 heartbeater.cc:511] Master 127.0.98.61:39897 requested a full tablet report, sending...
I20260926 13:14:33.387586 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45117
I20260926 13:14:33.387626 7593 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.387681 7593 heartbeater.cc:511] Master 127.0.98.62:45117 requested a full tablet report, sending...
I20260926 13:14:33.387912 6491 ts_manager.cc:194] Re-registered known tserver with Master: b290466c423a449b853ba9387ca51b2c (127.0.98.2:43917)
I20260926 13:14:33.387965 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43189
I20260926 13:14:33.388006 7592 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.387993 7088 ts_manager.cc:194] Registered new tserver with Master: b290466c423a449b853ba9387ca51b2c (127.0.98.2:43917)
I20260926 13:14:33.388072 7592 heartbeater.cc:511] Master 127.0.98.60:43189 requested a full tablet report, sending...
I20260926 13:14:33.388341 7276 ts_manager.cc:194] Registered new tserver with Master: b290466c423a449b853ba9387ca51b2c (127.0.98.2:43917)
I20260926 13:14:33.388473 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39245
W20260926 13:14:33.470111 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:14:33.470342 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:14:33.470366 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:14:33.471467 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:14:33.471503 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:14:33.471529 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:14:33.472918 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39811
--tserver_master_addrs=127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:14:33.473109 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:14:33.473286 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:14:33.478410 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:33.478403 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:14:33.478379 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:14:33.478650 7599 server_base.cc:1034] running on GCE node
I20260926 13:14:33.478895 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:14:33.479156 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:14:33.480334 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790428473480330 us; error 38 us; skew 500 ppm
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:14:33.482985 7599 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:14:33.483462 7599 webserver.cc:460] Webserver started at http://127.0.98.3:39811/ using document root <none> and password file <none>
I20260926 13:14:33.483702 7599 fs_manager.cc:359] Metadata directory not provided
I20260926 13:14:33.483793 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:14:33.485188 7599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:33.485780 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.486012 7599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:14:33.486140 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "73d7b08afc7b4173996a74341328a3ce"
format_stamp: "Formatted at 2026-09-26 13:14:32 on dist-test-slave-f864"
server_key: "12e07b621a936a60ab8568edf438a0c3"
server_key_iv: "32f5cf9883ef71a015970450804a74c7"
server_key_version: "encryptionkey@0"
I20260926 13:14:33.486563 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:14:33.504988 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:14:33.507241 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:14:33.507416 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:14:33.507663 7599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:14:33.507977 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:14:33.508031 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.508076 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:14:33.508100 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:14:33.514393 7599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35047
I20260926 13:14:33.514438 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35047 every 8 connection(s)
I20260926 13:14:33.514780 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:14:33.521972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 7599
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 13:14:33 dist-test-slave-f864 krb5kdc[6446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428473, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:14:33.528126 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.515652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39091 (local address 127.0.98.60:43189)
0926 13:14:33.515696 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.515698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.516046 (+ 348us) server_negotiation.cc:407] Connection header received
0926 13:14:33.516488 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.516489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.516501 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.516556 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.517443 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.517939 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.519598 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.519788 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.519859 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.524452 (+ 4593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.524453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.524454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.524461 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.526250 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.526936 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.526938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.526938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.526998 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.527287 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.527288 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.527289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.527439 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.527481 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.527924 (+ 443us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.528032 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:14:33.528644 6530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.515535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60617 (local address 127.0.98.62:45117)
0926 13:14:33.515589 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.515592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.515840 (+ 248us) server_negotiation.cc:407] Connection header received
0926 13:14:33.516501 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.516502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.516514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.516547 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.517448 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.518231 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.520806 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.521028 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.521114 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.524013 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.524014 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.524014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.524022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.526249 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.527013 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.527014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.527015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.527072 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.527288 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.527289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.527289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.527413 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.527460 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.528482 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.528555 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:33.528753 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.515522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43189 (local address 127.0.98.3:39091)
0926 13:14:33.515963 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.515973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.516387 (+ 414us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.516932 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.516933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.517129 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.517340 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.517344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.518045 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.518046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.519497 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.519500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.519878 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.519879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.520010 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.521192 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.521197 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.523583 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.526346 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.526352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.526355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.526805 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.527073 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.527074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.527074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.527211 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.527542 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.527548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.527758 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.528277 (+ 519us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.528471 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":70,"spinlock_wait_cycles":384,"thread_start_us":98,"threads_started":1}
I20260926 13:14:33.529572 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.515522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:60617)
0926 13:14:33.515689 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.515731 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.516381 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.516642 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.516647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.517124 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.517342 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.517347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.519600 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.519601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.520719 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.520721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.521116 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.521117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.521182 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.521563 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.521568 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.523892 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.526343 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.526352 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.526355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.526935 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.527142 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.527143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.527143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.527211 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.528063 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.528064 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.528104 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.529310 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.529428 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":19,"thread_start_us":96,"threads_started":1}
I20260926 13:14:33.529654 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43189
I20260926 13:14:33.530265 7728 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.530380 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45117
I20260926 13:14:33.530427 7729 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.530483 7728 heartbeater.cc:511] Master 127.0.98.60:43189 requested a full tablet report, sending...
I20260926 13:14:33.530500 7729 heartbeater.cc:511] Master 127.0.98.62:45117 requested a full tablet report, sending...
I20260926 13:14:33.530853 6491 ts_manager.cc:194] Re-registered known tserver with Master: 73d7b08afc7b4173996a74341328a3ce (127.0.98.3:35047)
I20260926 13:14:33.530929 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.515908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53361 (local address 127.0.98.61:39897)
0926 13:14:33.515952 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.515954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.516104 (+ 150us) server_negotiation.cc:407] Connection header received
0926 13:14:33.516501 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.516502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.516514 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.516547 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.517460 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.518224 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.519909 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.520086 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.520176 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.522625 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.522627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.522627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.522635 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.524320 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.529907 (+ 5587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.529909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.529909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.529977 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.530266 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.530267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.530267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.530415 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.530463 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.530756 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.530846 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:14:33.531188 7276 ts_manager.cc:194] Registered new tserver with Master: 73d7b08afc7b4173996a74341328a3ce (127.0.98.3:35047)
I20260926 13:14:33.531312 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.515848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39897 (local address 127.0.98.3:53361)
0926 13:14:33.516066 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.516072 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.516382 (+ 310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.516654 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.516654 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.517124 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:14:33.517342 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.517347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.518311 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.518311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.519497 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.519500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.520188 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.520190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.520260 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.520670 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:14:33.520680 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:14:33.522526 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:14:33.526477 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.526477 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.526478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.526836 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.530056 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:14:33.530057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:14:33.530058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:14:33.530122 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:14:33.530595 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:14:33.530596 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:14:33.530636 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.531094 (+ 458us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.531193 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":14,"thread_start_us":62,"threads_started":1}
I20260926 13:14:33.531340 6491 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60617
I20260926 13:14:33.531911 7730 heartbeater.cc:347] Connected to a master server at 127.0.98.61:39897
I20260926 13:14:33.531961 7730 heartbeater.cc:464] Registering TS with master...
I20260926 13:14:33.532032 7730 heartbeater.cc:511] Master 127.0.98.61:39897 requested a full tablet report, sending...
I20260926 13:14:33.532286 7088 ts_manager.cc:194] Registered new tserver with Master: 73d7b08afc7b4173996a74341328a3ce (127.0.98.3:35047)
I20260926 13:14:33.534298 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.523651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.0.98.61:39897)
0926 13:14:33.524566 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.524568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.524575 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:33.524595 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.524596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.524607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.525288 (+ 681us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.525321 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.526040 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.527002 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.527293 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.527595 (+ 302us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.529026 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.529027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.529028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.529034 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.532870 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.533351 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.533352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.533352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.533414 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.533622 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.533624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.533624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.533762 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.533903 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.534060 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.534139 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898,"thread_start_us":52,"threads_started":1}
I20260926 13:14:33.535431 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.523871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.0.98.60:43189)
0926 13:14:33.527671 (+ 3800us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.527678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.527685 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:14:33.527727 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.527727 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.527739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.528471 (+ 732us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.528510 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.529255 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.530314 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.530586 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.530933 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.532242 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.532243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.532243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.532250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.534127 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.534665 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.534666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.534666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.534730 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.534939 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.534940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.534940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.535049 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.535146 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.535249 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.535326 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3778,"thread_start_us":57,"threads_started":1}
I20260926 13:14:33.536621 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.523083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.0.98.62:45117)
0926 13:14:33.524124 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.524126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.524131 (+ 5us) server_negotiation.cc:407] Connection header received
0926 13:14:33.524148 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.524149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.524158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.524234 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.525016 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.525828 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.526958 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.527291 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.527373 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.533530 (+ 6157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.533532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.533532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.533539 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.535184 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.535810 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.535811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.535812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.535869 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.536081 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.536082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.536083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.536195 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.536300 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.536397 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.536482 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":80,"threads_started":1}
I20260926 13:14:33.537995 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:14:33.539677 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:14:33.547878 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.544094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35047 (local address 127.0.0.1:39520)
0926 13:14:33.544242 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.544251 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.544292 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.544624 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.544624 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.544627 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:33.544673 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.544675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.545922 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.545924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.546854 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.546856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.547666 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.547667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.547712 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.547715 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.547749 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.547780 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 13:14:33.547885 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.544184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.0.98.3:35047)
0926 13:14:33.544339 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.544345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.544355 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:33.544443 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.544445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.544463 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.544547 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:33.544750 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.545818 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.546939 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.547588 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.547667 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.547743 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.547770 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.547807 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260926 13:14:33.548269 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.543981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33261 (local address 127.0.0.1:42874)
0926 13:14:33.544052 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.544066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.544115 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.544565 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.544566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.544568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:33.544618 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.544620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.545977 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.545977 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.547152 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.547154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.548067 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.548068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.548123 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.548126 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.548166 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.548207 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":45}
I20260926 13:14:33.548314 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.544089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.0.98.1:33261)
0926 13:14:33.544394 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.544398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.544406 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:14:33.544443 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.544445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.544463 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.544564 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:33.544683 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.545845 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.547243 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.547943 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.548016 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.548098 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.548193 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.548235 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260926 13:14:33.549165 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.544297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43917 (local address 127.0.0.1:43906)
0926 13:14:33.544398 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.544409 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.544465 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.544977 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.544978 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.544980 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:33.545020 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.545021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.545865 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.545866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.547152 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.547154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.548932 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.548934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.548992 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.548997 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.549055 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.549093 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":37,"thread_start_us":57,"threads_started":1}
I20260926 13:14:33.549185 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.544511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.0.98.2:43917)
0926 13:14:33.544757 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.544762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.544772 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:14:33.544812 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.544813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.544828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.544902 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:33.545103 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.545778 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.548134 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.548820 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.548910 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.548980 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.549071 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.549115 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":85,"threads_started":1}
I20260926 13:14:33.549175 7389 tablet_service.cc:1467] Processing CreateTablet for tablet cc6879a2f9ac4133bea8f2fb194bc03a (DEFAULT_TABLE table=test-table [id=395c3c629ccc4d6d8fc503a7c8f8587c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:33.549597 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6879a2f9ac4133bea8f2fb194bc03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:33.549077 7662 tablet_service.cc:1467] Processing CreateTablet for tablet cc6879a2f9ac4133bea8f2fb194bc03a (DEFAULT_TABLE table=test-table [id=395c3c629ccc4d6d8fc503a7c8f8587c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:33.549901 7662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6879a2f9ac4133bea8f2fb194bc03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:33.550103 7526 tablet_service.cc:1467] Processing CreateTablet for tablet cc6879a2f9ac4133bea8f2fb194bc03a (DEFAULT_TABLE table=test-table [id=395c3c629ccc4d6d8fc503a7c8f8587c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:14:33.550479 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6879a2f9ac4133bea8f2fb194bc03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:14:33.552829 7754 tablet_bootstrap.cc:492] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Bootstrap starting.
I20260926 13:14:33.553525 7754 tablet_bootstrap.cc:654] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:33.553634 7755 tablet_bootstrap.cc:492] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Bootstrap starting.
I20260926 13:14:33.553881 7754 log.cc:824] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:33.554153 7756 tablet_bootstrap.cc:492] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Bootstrap starting.
I20260926 13:14:33.554330 7755 tablet_bootstrap.cc:654] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:33.554654 7755 log.cc:824] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:33.554832 7754 tablet_bootstrap.cc:492] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: No bootstrap required, opened a new log
I20260926 13:14:33.554858 7756 tablet_bootstrap.cc:654] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Neither blocks nor log segments found. Creating new log.
I20260926 13:14:33.554906 7754 ts_tablet_manager.cc:1397] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:14:33.555154 7756 log.cc:824] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:14:33.555913 7755 tablet_bootstrap.cc:492] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: No bootstrap required, opened a new log
I20260926 13:14:33.555975 7755 ts_tablet_manager.cc:1397] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:14:33.555984 7756 tablet_bootstrap.cc:492] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: No bootstrap required, opened a new log
I20260926 13:14:33.556046 7756 ts_tablet_manager.cc:1397] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:14:33.557327 7754 raft_consensus.cc:348] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.557502 7754 raft_consensus.cc:374] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:33.557549 7754 raft_consensus.cc:729] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d7b08afc7b4173996a74341328a3ce, State: Initialized, Role: FOLLOWER
I20260926 13:14:33.557667 7754 consensus_queue.cc:260] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.557984 7754 ts_tablet_manager.cc:1428] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:14:33.558131 7729 heartbeater.cc:503] Master 127.0.98.62:45117 was elected leader, sending a full tablet report...
I20260926 13:14:33.558617 7755 raft_consensus.cc:348] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.558777 7755 raft_consensus.cc:374] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:33.558821 7755 raft_consensus.cc:729] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc44d8c907bf4f898a3aa007a9a3b9e5, State: Initialized, Role: FOLLOWER
I20260926 13:14:33.558995 7755 consensus_queue.cc:260] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.559306 7456 heartbeater.cc:503] Master 127.0.98.62:45117 was elected leader, sending a full tablet report...
I20260926 13:14:33.559291 7755 ts_tablet_manager.cc:1428] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 13:14:33.559412 7756 raft_consensus.cc:348] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.559576 7756 raft_consensus.cc:374] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:14:33.559616 7756 raft_consensus.cc:729] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b290466c423a449b853ba9387ca51b2c, State: Initialized, Role: FOLLOWER
I20260926 13:14:33.559902 7756 consensus_queue.cc:260] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.560178 7756 ts_tablet_manager.cc:1428] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 13:14:33.560499 7593 heartbeater.cc:503] Master 127.0.98.62:45117 was elected leader, sending a full tablet report...
I20260926 13:14:33.617172 7762 raft_consensus.cc:482] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:14:33.617413 7762 raft_consensus.cc:504] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.617897 7762 leader_election.cc:288] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc44d8c907bf4f898a3aa007a9a3b9e5 (127.0.98.1:33261), 73d7b08afc7b4173996a74341328a3ce (127.0.98.3:35047)
I20260926 13:14:33.620828 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.618197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40997 (local address 127.0.98.1:33261)
0926 13:14:33.618276 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.618278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.618290 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:14:33.618323 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.618324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.618337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.618437 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:33.618679 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.619181 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.620128 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.620478 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.620533 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.620542 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.620706 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.620758 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:14:33.620828 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.618099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33261 (local address 127.0.98.2:40997)
0926 13:14:33.618183 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.618196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.618247 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.618509 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.618510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.618512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:33.618590 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.618592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.619286 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.619287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.620021 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.620023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.620553 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.620554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.620633 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.620636 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.620693 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.620726 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 13:14:33.620895 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.618369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46927 (local address 127.0.98.3:35047)
0926 13:14:33.618433 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.618435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.618443 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:14:33.618470 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.618471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.618483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.618541 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:14:33.618727 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.619171 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.620123 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.620535 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.620605 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.620615 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.620780 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.620820 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:14:33.620947 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.618275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35047 (local address 127.0.98.2:46927)
0926 13:14:33.618316 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:33.618325 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:14:33.618355 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:14:33.618561 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:14:33.618561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:14:33.618563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:14:33.618648 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.618649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.619261 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.619262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.620023 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:14:33.620025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.620621 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.620621 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:14:33.620703 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.620706 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:14:33.620763 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:14:33.620803 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":6}
I20260926 13:14:33.621277 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6879a2f9ac4133bea8f2fb194bc03a" candidate_uuid: "b290466c423a449b853ba9387ca51b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" is_pre_election: true
I20260926 13:14:33.621474 7409 raft_consensus.cc:2454] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b290466c423a449b853ba9387ca51b2c in term 0.
I20260926 13:14:33.621410 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6879a2f9ac4133bea8f2fb194bc03a" candidate_uuid: "b290466c423a449b853ba9387ca51b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d7b08afc7b4173996a74341328a3ce" is_pre_election: true
I20260926 13:14:33.621585 7682 raft_consensus.cc:2454] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b290466c423a449b853ba9387ca51b2c in term 0.
I20260926 13:14:33.621730 7482 leader_election.cc:302] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b290466c423a449b853ba9387ca51b2c, bc44d8c907bf4f898a3aa007a9a3b9e5; no voters:
I20260926 13:14:33.621891 7762 raft_consensus.cc:2781] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:14:33.621970 7762 raft_consensus.cc:482] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:14:33.622005 7762 raft_consensus.cc:3037] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:33.622951 7762 raft_consensus.cc:504] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.623095 7762 leader_election.cc:288] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [CANDIDATE]: Term 1 election: Requested vote from peers bc44d8c907bf4f898a3aa007a9a3b9e5 (127.0.98.1:33261), 73d7b08afc7b4173996a74341328a3ce (127.0.98.3:35047)
I20260926 13:14:33.623273 7682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6879a2f9ac4133bea8f2fb194bc03a" candidate_uuid: "b290466c423a449b853ba9387ca51b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73d7b08afc7b4173996a74341328a3ce"
I20260926 13:14:33.623260 7409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6879a2f9ac4133bea8f2fb194bc03a" candidate_uuid: "b290466c423a449b853ba9387ca51b2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5"
I20260926 13:14:33.623364 7682 raft_consensus.cc:3037] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:14:33.623361 7409 raft_consensus.cc:3037] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 0 FOLLOWER]: Advancing to term 1
W20260926 13:14:33.623715 7595 tablet.cc:2369] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:14:33.624159 7682 raft_consensus.cc:2454] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b290466c423a449b853ba9387ca51b2c in term 1.
I20260926 13:14:33.624274 7409 raft_consensus.cc:2454] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b290466c423a449b853ba9387ca51b2c in term 1.
I20260926 13:14:33.624393 7482 leader_election.cc:302] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d7b08afc7b4173996a74341328a3ce, b290466c423a449b853ba9387ca51b2c; no voters:
I20260926 13:14:33.624482 7762 raft_consensus.cc:2781] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:14:33.624624 7762 raft_consensus.cc:686] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 1 LEADER]: Becoming Leader. State: Replica: b290466c423a449b853ba9387ca51b2c, State: Running, Role: LEADER
I20260926 13:14:33.624722 7762 consensus_queue.cc:237] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } }
I20260926 13:14:33.625976 6490 catalog_manager.cc:5495] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c reported cstate change: term changed from 0 to 1, leader changed from <none> to b290466c423a449b853ba9387ca51b2c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b290466c423a449b853ba9387ca51b2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b290466c423a449b853ba9387ca51b2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43917 } health_report { overall_health: HEALTHY } } }
I20260926 13:14:33.645959 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.643520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.0.98.2:43917)
0926 13:14:33.643575 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.643578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.643607 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:14:33.643649 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.643650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.643670 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.643746 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:14:33.643900 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.644379 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.645142 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.645374 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.645449 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.645626 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:14:33.645727 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:14:33.645824 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.645842 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.645867 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:14:33.647830 7409 raft_consensus.cc:1261] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 1 FOLLOWER]: Refusing update from remote peer b290466c423a449b853ba9387ca51b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:33.647830 7682 raft_consensus.cc:1261] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 1 FOLLOWER]: Refusing update from remote peer b290466c423a449b853ba9387ca51b2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:14:33.648212 7762 consensus_queue.cc:1035] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc44d8c907bf4f898a3aa007a9a3b9e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:33.648377 7763 consensus_queue.cc:1035] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73d7b08afc7b4173996a74341328a3ce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:14:33.651742 7768 mvcc.cc:205] Tried to move back new op lower bound from 7333595028058533888 to 7333595027967926272. Current Snapshot: MvccSnapshot[applied={T|T < 7333595028058533888}]
I20260926 13:14:33.653425 7769 mvcc.cc:205] Tried to move back new op lower bound from 7333595028058533888 to 7333595027967926272. Current Snapshot: MvccSnapshot[applied={T|T < 7333595028058533888}]
I20260926 13:14:33.654312 6490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:33.654431 6490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:14:33.656132 6490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25e06fc9a29a471182200dfd21afc633: Sending DeleteTablet for 3 replicas of tablet cc6879a2f9ac4133bea8f2fb194bc03a
I20260926 13:14:33.656507 7662 tablet_service.cc:1514] Processing DeleteTablet for tablet cc6879a2f9ac4133bea8f2fb194bc03a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39520
I20260926 13:14:33.656481 7389 tablet_service.cc:1514] Processing DeleteTablet for tablet cc6879a2f9ac4133bea8f2fb194bc03a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42874
I20260926 13:14:33.657184 7778 tablet_replica.cc:330] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: stopping tablet replica
I20260926 13:14:33.657198 7526 tablet_service.cc:1514] Processing DeleteTablet for tablet cc6879a2f9ac4133bea8f2fb194bc03a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:14:33 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43906
I20260926 13:14:33.657389 7778 raft_consensus.cc:2229] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:33.657445 7779 tablet_replica.cc:330] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: stopping tablet replica
I20260926 13:14:33.657545 7779 raft_consensus.cc:2229] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:14:33.657629 7778 raft_consensus.cc:2258] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:33.657652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6459
I20260926 13:14:33.658612 7779 raft_consensus.cc:2258] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:33.658882 7777 tablet_replica.cc:330] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: stopping tablet replica
I20260926 13:14:33.658991 7777 raft_consensus.cc:2229] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:14:33.659092 7777 raft_consensus.cc:2258] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:14:33.659742 7779 ts_tablet_manager.cc:1905] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:33.661461 7779 ts_tablet_manager.cc:1918] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:33.661537 7779 log.cc:1196] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/cc6879a2f9ac4133bea8f2fb194bc03a
I20260926 13:14:33.661823 7779 ts_tablet_manager.cc:1939] T cc6879a2f9ac4133bea8f2fb194bc03a P b290466c423a449b853ba9387ca51b2c: Deleting consensus metadata
I20260926 13:14:33.662401 7777 ts_tablet_manager.cc:1905] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:14:33.663614 7479 connection.cc:535] server connection from 127.0.0.1:43906 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:14:33.663983 7777 ts_tablet_manager.cc:1918] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:33.664057 7777 log.cc:1196] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/cc6879a2f9ac4133bea8f2fb194bc03a
I20260926 13:14:33.664327 7777 ts_tablet_manager.cc:1939] T cc6879a2f9ac4133bea8f2fb194bc03a P bc44d8c907bf4f898a3aa007a9a3b9e5: Deleting consensus metadata
W20260926 13:14:33.664789 7343 connection.cc:441] server connection from 127.0.0.1:42874 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42874 (request call id 1) could send its response
I20260926 13:14:33.664992 7778 ts_tablet_manager.cc:1905] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:14:33.666755 7778 ts_tablet_manager.cc:1918] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:14:33.666879 7778 log.cc:1196] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/cc6879a2f9ac4133bea8f2fb194bc03a
I20260926 13:14:33.667243 7778 ts_tablet_manager.cc:1939] T cc6879a2f9ac4133bea8f2fb194bc03a P 73d7b08afc7b4173996a74341328a3ce: Deleting consensus metadata
W20260926 13:14:33.667639 7618 connection.cc:441] server connection from 127.0.0.1:39520 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39520 (request call id 1) could send its response
I20260926 13:14:33.676035 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.657610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.0.98.60:43189)
0926 13:14:33.657655 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.657657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.665863 (+ 8206us) server_negotiation.cc:407] Connection header received
0926 13:14:33.665979 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.665981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.665996 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.666029 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.666937 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.667892 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.669114 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.669503 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.669596 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.671484 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.671485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.671486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.671495 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.674196 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.674624 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.674626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.674627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.674686 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.674944 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.674945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.674945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.675084 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.675184 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.675407 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.675919 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:14:33.677034 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.665588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.0.98.60:43189)
0926 13:14:33.665630 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.665632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.665965 (+ 333us) server_negotiation.cc:407] Connection header received
0926 13:14:33.666052 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.666053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.666061 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.666090 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.669690 (+ 3600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.670655 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.671812 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.672133 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.672177 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.673693 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.673695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.673695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.673703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.675757 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.676179 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.676180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.676180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.676240 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.676456 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.676456 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.676457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.676584 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.676655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.676900 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.676977 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:14:33.677810 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:33.665682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.98.61:39897)
0926 13:14:33.665729 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:14:33.665731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:14:33.665949 (+ 218us) server_negotiation.cc:407] Connection header received
0926 13:14:33.666056 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:14:33.666057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:14:33.666082 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:14:33.666116 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:14:33.670149 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.670949 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.672041 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:14:33.672362 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:14:33.672415 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:14:33.674228 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:14:33.674229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:14:33.674230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:14:33.674240 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:14:33.676325 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.676760 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.676761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.676762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.676820 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:14:33.677031 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:14:33.677031 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:14:33.677032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:14:33.677165 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:14:33.677239 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:14:33.677628 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 13:14:33.677727 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:14:34.653189 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:34.652853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:57175)
0926 13:14:34.653035 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:34.653130 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
I20260926 13:14:34.653189 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:34.652951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:47599)
0926 13:14:34.653086 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:34.653147 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":47,"threads_started":1}
W20260926 13:14:34.653493 7593 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
W20260926 13:14:34.653571 7456 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:34.654054 7790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:34.653743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:56845)
0926 13:14:34.653881 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:34.653959 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
W20260926 13:14:34.654301 7729 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45117 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:35.156585 7791 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:35.156713 7791 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:35.156926 7791 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:35.157318 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:35.157109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:55238)
0926 13:14:35.157222 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:35.157280 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
W20260926 13:14:35.157498 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:35.157918 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:35.157787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:55250)
0926 13:14:35.157862 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:35.157895 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 13:14:35.158023 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:35.158133 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:35.158279 7791 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:35.654228 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:35.653930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:54857)
0926 13:14:35.654101 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:35.654148 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260926 13:14:35.654266 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:35.653933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:40765)
0926 13:14:35.654139 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:35.654211 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260926 13:14:35.654894 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:35.654625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:51159)
0926 13:14:35.654810 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:35.654849 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260926 13:14:36.655048 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:36.654810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:58329)
0926 13:14:36.654937 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:36.654983 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260926 13:14:36.655503 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:36.655315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:58511)
0926 13:14:36.655415 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:36.655450 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":46,"threads_started":1}
I20260926 13:14:36.655733 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:36.654752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:58393)
0926 13:14:36.655647 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:36.655685 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":83,"threads_started":1}
I20260926 13:14:37.215000 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:37.215099 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:37.215219 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:37.215610 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:37.215416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:49986)
0926 13:14:37.215525 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:37.215570 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260926 13:14:37.216028 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:37.215932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:49998)
0926 13:14:37.215981 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:37.216009 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:14:37.216110 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:37.216185 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:37.216274 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:37.656113 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:37.655857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:49407)
0926 13:14:37.656007 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:37.656057 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260926 13:14:37.656179 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:37.655601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:60677)
0926 13:14:37.656085 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:37.656129 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":81,"threads_started":1}
I20260926 13:14:37.656570 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:37.656402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:50715)
0926 13:14:37.656509 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:37.656539 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":50,"threads_started":1}
I20260926 13:14:38.656860 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:38.656611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:51197)
0926 13:14:38.656760 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:38.656805 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":53,"threads_started":1}
I20260926 13:14:38.657052 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:38.656595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:56713)
0926 13:14:38.656958 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:38.657004 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":68,"threads_started":1}
I20260926 13:14:38.657385 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:38.657044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:42571)
0926 13:14:38.657311 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:38.657344 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":51,"threads_started":1}
I20260926 13:14:39.657881 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:39.657524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:37601)
0926 13:14:39.657740 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:39.657821 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":65,"threads_started":1}
I20260926 13:14:39.658208 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:39.657575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:58993)
0926 13:14:39.658075 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:39.658116 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":68,"threads_started":1}
I20260926 13:14:39.658406 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:39.657845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:47825)
0926 13:14:39.658299 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:39.658337 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":44,"threads_started":1}
I20260926 13:14:40.146862 7812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:40.146962 7812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:40.147130 7812 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:40.147490 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:40.147288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:50344)
0926 13:14:40.147406 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:40.147451 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260926 13:14:40.148032 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:40.147909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:50352)
0926 13:14:40.147974 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:40.148006 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 13:14:40.148141 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:40.148231 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:40.148310 7812 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:40.658795 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:40.658501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:37123)
0926 13:14:40.658656 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:40.658701 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260926 13:14:40.659170 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:40.658962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:55405)
0926 13:14:40.659080 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:40.659111 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20260926 13:14:40.659339 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:40.658957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:36489)
0926 13:14:40.659247 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:40.659287 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":53,"threads_started":1}
I20260926 13:14:41.659703 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:41.659381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:45433)
0926 13:14:41.659593 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:41.659638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":73,"threads_started":1}
I20260926 13:14:41.660218 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:41.659788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:44005)
0926 13:14:41.660101 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:41.660140 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":66,"threads_started":1}
I20260926 13:14:41.660243 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:41.660025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:50839)
0926 13:14:41.660164 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:41.660199 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":55,"threads_started":1}
I20260926 13:14:42.660939 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:42.660388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:49813)
0926 13:14:42.660827 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:42.660877 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":73,"threads_started":1}
I20260926 13:14:42.660959 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:42.660655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:51581)
0926 13:14:42.660870 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:42.660909 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":51,"threads_started":1}
I20260926 13:14:42.661198 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:42.660639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:60429)
0926 13:14:42.661130 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:42.661163 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":77,"threads_started":1}
I20260926 13:14:42.709643 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:42.709756 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:42.709915 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:42.710498 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:42.710098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:50536)
0926 13:14:42.710407 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:42.710459 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":103,"threads_started":1}
W20260926 13:14:42.710654 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260926 13:14:42.711149 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:42.711016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:50540)
0926 13:14:42.711072 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:42.711120 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 13:14:42.711279 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:42.711336 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:42.711405 7824 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:43.661881 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:43.661485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:38077)
0926 13:14:43.661617 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:43.661668 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260926 13:14:43.661942 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:43.661695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:45337)
0926 13:14:43.661845 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:43.661903 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":41,"threads_started":1}
I20260926 13:14:43.661972 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:43.661607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:41081)
0926 13:14:43.661724 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:43.661774 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260926 13:14:44.662663 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:44.662376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:52023)
0926 13:14:44.662551 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:44.662602 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":67,"threads_started":1}
I20260926 13:14:44.662660 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:44.662479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:55167)
0926 13:14:44.662569 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:44.662606 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":46,"threads_started":1}
I20260926 13:14:44.662871 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:44.662451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:46391)
0926 13:14:44.662760 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:44.662833 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":77,"threads_started":1}
I20260926 13:14:45.663509 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:45.663139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:40477)
0926 13:14:45.663385 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:45.663440 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260926 13:14:45.663794 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:45.663139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:54209)
0926 13:14:45.663654 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:45.663751 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":85,"threads_started":1}
I20260926 13:14:45.663897 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:45.663532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:47003)
0926 13:14:45.663771 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:45.663851 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":58,"threads_started":1}
I20260926 13:14:46.664536 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:46.664286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:39943)
0926 13:14:46.664429 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:46.664475 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260926 13:14:46.664759 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:46.664418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:56949)
0926 13:14:46.664677 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:46.664714 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":43,"threads_started":1}
I20260926 13:14:46.664870 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:46.664295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:39205)
0926 13:14:46.664777 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:46.664828 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":60,"threads_started":1}
I20260926 13:14:47.665318 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:47.665068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:34465)
0926 13:14:47.665205 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:47.665252 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260926 13:14:47.665858 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:47.665382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:47063)
0926 13:14:47.665751 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:47.665808 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":44,"threads_started":1}
I20260926 13:14:47.665872 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:47.665249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:55791)
0926 13:14:47.665779 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:47.665819 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":53,"threads_started":1}
I20260926 13:14:48.666370 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:48.666099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:50805)
0926 13:14:48.666270 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:48.666313 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":61,"threads_started":1}
I20260926 13:14:48.666803 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:48.666428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:37591)
0926 13:14:48.666721 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:48.666760 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":46,"threads_started":1}
I20260926 13:14:48.666832 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:48.666258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:50271)
0926 13:14:48.666761 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:48.666792 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":54,"threads_started":1}
I20260926 13:14:49.667135 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:49.666872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:48907)
0926 13:14:49.667025 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:49.667074 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260926 13:14:49.667692 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:49.667421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:51629)
0926 13:14:49.667620 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:49.667660 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260926 13:14:49.667691 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:49.667239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:40059)
0926 13:14:49.667600 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:49.667638 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":50,"threads_started":1}
I20260926 13:14:50.119480 7848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:50.119642 7848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:50.119829 7848 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:50.120214 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:50.120017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54350)
0926 13:14:50.120123 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:50.120174 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":55,"threads_started":1}
W20260926 13:14:50.120306 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:50.120683 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:50.120577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54360)
0926 13:14:50.120634 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:50.120665 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 13:14:50.120761 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:50.120822 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:50.120911 7848 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:50.668143 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:50.667791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:55739)
0926 13:14:50.668027 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:50.668077 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
I20260926 13:14:50.668493 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:50.668250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:44155)
0926 13:14:50.668411 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:50.668445 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":59,"threads_started":1}
I20260926 13:14:50.668732 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:50.668438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:54111)
0926 13:14:50.668656 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:50.668700 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260926 13:14:51.669312 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:51.669116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:39839)
0926 13:14:51.669219 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:51.669261 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":47,"threads_started":1}
I20260926 13:14:51.669312 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:51.669019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:50105)
0926 13:14:51.669233 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:51.669261 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":66,"threads_started":1}
I20260926 13:14:51.669406 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:51.669145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:37939)
0926 13:14:51.669317 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:51.669356 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":57,"threads_started":1}
I20260926 13:14:52.252462 7857 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:52.252590 7857 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:52.252795 7857 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:52.253257 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:52.253005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54470)
0926 13:14:52.253139 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:52.253181 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260926 13:14:52.253685 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:52.253594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:54484)
0926 13:14:52.253641 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:52.253667 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:14:52.253770 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:52.253854 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:52.253979 7857 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:52.670130 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:52.669908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:33033)
0926 13:14:52.670064 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:52.670090 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":55,"threads_started":1}
I20260926 13:14:52.670130 7860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:52.669804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:53469)
0926 13:14:52.670032 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:52.670074 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":66,"threads_started":1}
I20260926 13:14:52.670150 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:52.669806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:50311)
0926 13:14:52.670048 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:52.670093 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":88,"threads_started":1}
I20260926 13:14:53.670949 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:53.670587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:40685)
0926 13:14:53.670839 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:53.670889 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":72,"threads_started":1}
I20260926 13:14:53.671161 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:53.670856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:54091)
0926 13:14:53.671046 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:53.671080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":45,"threads_started":1}
I20260926 13:14:53.671267 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:53.670587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:47769)
0926 13:14:53.671156 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:53.671196 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":68,"threads_started":1}
I20260926 13:14:54.671844 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:54.671536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:60257)
0926 13:14:54.671742 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:54.671788 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":63,"threads_started":1}
I20260926 13:14:54.671991 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:54.671709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:48667)
0926 13:14:54.671903 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:54.671940 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":66,"threads_started":1}
I20260926 13:14:54.672079 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:54.671728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:52201)
0926 13:14:54.672002 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:54.672036 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":53,"threads_started":1}
I20260926 13:14:55.672793 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:55.672464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:34305)
0926 13:14:55.672707 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:55.672745 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":65,"threads_started":1}
I20260926 13:14:55.672796 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:55.672559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:47373)
0926 13:14:55.672666 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:55.672732 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":43,"threads_started":1}
I20260926 13:14:55.673147 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:55.672748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:33619)
0926 13:14:55.673070 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:55.673113 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":60,"threads_started":1}
I20260926 13:14:56.673553 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:56.673322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:40967)
0926 13:14:56.673450 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:56.673496 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260926 13:14:56.673595 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:56.673364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:59475)
0926 13:14:56.673528 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:56.673555 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":55,"threads_started":1}
I20260926 13:14:56.673954 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:56.673605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:43375)
0926 13:14:56.673868 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:56.673907 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":47,"threads_started":1}
I20260926 13:14:57.674530 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:57.674309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:39077)
0926 13:14:57.674417 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:57.674467 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260926 13:14:57.674655 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:57.674014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:55431)
0926 13:14:57.674556 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:57.674602 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":126,"threads_started":1}
I20260926 13:14:57.674897 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:57.674588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:52111)
0926 13:14:57.674825 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:57.674855 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":171,"threads_started":1}
I20260926 13:14:58.675333 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:58.675010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:47299)
0926 13:14:58.675229 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:58.675272 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":69,"threads_started":1}
I20260926 13:14:58.675503 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:58.675240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:38913)
0926 13:14:58.675421 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:58.675457 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":51,"threads_started":1}
I20260926 13:14:58.675658 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:58.675364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:44067)
0926 13:14:58.675580 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:58.675613 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":49,"threads_started":1}
I20260926 13:14:59.635612 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 25e06fc9a29a471182200dfd21afc633)
I20260926 13:14:59.635756 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "25e06fc9a29a471182200dfd21afc633" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45117 } } peers { permanent_uuid: "20c3714dbe9e4b50b61b94d47ca0f762" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39897 } attrs { promote: false } } peers { permanent_uuid: "5669b0b0812e472f95acaa3fb78f6561" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43189 } attrs { promote: true } }
I20260926 13:14:59.635883 7881 leader_election.cc:288] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117)
I20260926 13:14:59.636332 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:59.636126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:33602)
0926 13:14:59.636244 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:59.636292 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
W20260926 13:14:59.636444 7075 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20260926 13:14:59.636878 7882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:59.636736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.0.1:33612)
0926 13:14:59.636809 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:59.636852 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 13:14:59.636977 7075 leader_election.cc:334] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 25e06fc9a29a471182200dfd21afc633 (127.0.98.62:45117): Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
I20260926 13:14:59.637049 7075 leader_election.cc:302] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 20c3714dbe9e4b50b61b94d47ca0f762; no voters: 25e06fc9a29a471182200dfd21afc633
I20260926 13:14:59.637138 7881 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 20c3714dbe9e4b50b61b94d47ca0f762 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:14:59.676245 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:59.675796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:52971)
0926 13:14:59.676138 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:59.676187 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":85,"threads_started":1}
I20260926 13:14:59.676301 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:59.676069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:39265)
0926 13:14:59.676211 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:59.676249 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":53,"threads_started":1}
I20260926 13:14:59.676826 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:14:59.676632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:41763)
0926 13:14:59.676765 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 13:14:59.676791 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260926 13:15:00.677073 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:00.676805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:45779)
0926 13:15:00.676971 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:00.677014 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":65,"threads_started":1}
I20260926 13:15:00.677191 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:00.676805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:38411)
0926 13:15:00.677099 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:00.677137 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":70,"threads_started":1}
I20260926 13:15:00.677588 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:00.677361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:54099)
0926 13:15:00.677484 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:00.677520 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":48,"threads_started":1}
I20260926 13:15:01.677978 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:01.677593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:60387)
0926 13:15:01.677831 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:01.677915 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":63,"threads_started":1}
I20260926 13:15:01.678467 7892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:01.678011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:50501)
0926 13:15:01.678390 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:01.678426 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":50,"threads_started":1}
I20260926 13:15:01.678488 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:01.677709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:44441)
0926 13:15:01.678388 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:01.678429 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":78,"threads_started":1}
I20260926 13:15:02.678866 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:02.678544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:34947)
0926 13:15:02.678732 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:02.678803 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":79,"threads_started":1}
I20260926 13:15:02.679167 7895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:02.678917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:54499)
0926 13:15:02.679085 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:02.679122 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":61,"threads_started":1}
I20260926 13:15:02.679390 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:02.678908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:53651)
0926 13:15:02.679299 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:02.679336 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":52,"threads_started":1}
I20260926 13:15:03.679780 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:03.679509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.1:45991)
0926 13:15:03.679672 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:03.679718 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":60,"threads_started":1}
I20260926 13:15:03.680289 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:03.679880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.2:60601)
0926 13:15:03.680154 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:03.680190 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":85,"threads_started":1}
I20260926 13:15:03.680286 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:03.680048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45117 (local address 127.0.98.3:41199)
0926 13:15:03.680197 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:03.680239 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":49,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.98.62:45117,127.0.98.61:39897,127.0.98.60:43189, num_attempts: 1)
I20260926 13:15:03.886075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 7235
I20260926 13:15:03.890939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 7463
I20260926 13:15:03.896275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 7599
I20260926 13:15:03.901096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 6955
I20260926 13:15:03.905645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 7142
2026-09-26T13:15:03Z chronyd exiting
I20260926 13:15:03.922113 392 test_util.cc:175] -----------------------------------------------
I20260926 13:15:03.922333 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790428444292953-392-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33179 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:15:03 dist-test-slave-f864 krb5kdc[7901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:15:03 dist-test-slave-f864 krb5kdc[7901](info): set up 2 sockets
Sep 26 13:15:03 dist-test-slave-f864 krb5kdc[7901](info): commencing operation
krb5kdc: starting...
W20260926 13:15:04.946763 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:15:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 13:15:05.065424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33881
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:05.145529 7914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:05.145774 7914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:05.145812 7914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:05.147259 7914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:05.147347 7914 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 13:15:05.147372 7914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:05.147401 7914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:05.147431 7914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:05.147460 7914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:05.148952 7914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33881
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:05.149256 7914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:05.149503 7914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:05.154489 7922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.154585 7914 server_base.cc:1034] running on GCE node
W20260926 13:15:05.154495 7920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:05.154497 7919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.154966 7914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:05.155228 7914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:05.156368 7914 hybrid_clock.cc:648] HybridClock initialized: now 1790428505156347 us; error 45 us; skew 500 ppm
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:05.158814 7914 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 13:15:05.159197 7914 webserver.cc:460] Webserver started at http://127.0.98.62:38537/ using document root <none> and password file <none>
I20260926 13:15:05.159443 7914 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:05.159516 7914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:05.159683 7914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:05.160774 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "732c2667b42d4077a702d43bc1f69ebe"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "6616a36ffeace4fc9c146072ac7fe780"
server_key_iv: "84829b031814c759e0e2c8651f752e7b"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.161202 7914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "732c2667b42d4077a702d43bc1f69ebe"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "6616a36ffeace4fc9c146072ac7fe780"
server_key_iv: "84829b031814c759e0e2c8651f752e7b"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.162640 7914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:15:05.163360 7929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.163535 7914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:05.163635 7914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "732c2667b42d4077a702d43bc1f69ebe"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "6616a36ffeace4fc9c146072ac7fe780"
server_key_iv: "84829b031814c759e0e2c8651f752e7b"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.163712 7914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:05.174751 7914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:05.176661 7914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:05.176895 7914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:05.181737 7914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33881
I20260926 13:15:05.181771 7991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33881 every 8 connection(s)
I20260926 13:15:05.182121 7914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:15:05.182878 7992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.185189 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Bootstrap starting.
I20260926 13:15:05.185804 7992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.186100 7992 log.cc:824] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:05.186820 7992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: No bootstrap required, opened a new log
I20260926 13:15:05.188942 7992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732c2667b42d4077a702d43bc1f69ebe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33881 } }
I20260926 13:15:05.189129 7992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.189179 7992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 732c2667b42d4077a702d43bc1f69ebe, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.189344 7992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732c2667b42d4077a702d43bc1f69ebe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33881 } }
I20260926 13:15:05.189433 7992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:15:05.189481 7992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:15:05.189534 7992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:05.190124 7992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732c2667b42d4077a702d43bc1f69ebe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33881 } }
I20260926 13:15:05.190276 7992 leader_election.cc:302] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 732c2667b42d4077a702d43bc1f69ebe; no voters:
I20260926 13:15:05.190412 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 7914
I20260926 13:15:05.190518 7992 leader_election.cc:288] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:15:05.190550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:15:05.190593 7995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:05.190773 7995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [term 1 LEADER]: Becoming Leader. State: Replica: 732c2667b42d4077a702d43bc1f69ebe, State: Running, Role: LEADER
I20260926 13:15:05.190790 7992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:15:05.190815 392 external_mini_cluster.cc:1428] Setting key 4c3c8945d486ced6b63e4a588655cdaa
I20260926 13:15:05.190871 7995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732c2667b42d4077a702d43bc1f69ebe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33881 } }
I20260926 13:15:05.191258 7997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 732c2667b42d4077a702d43bc1f69ebe. Latest consensus state: current_term: 1 leader_uuid: "732c2667b42d4077a702d43bc1f69ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732c2667b42d4077a702d43bc1f69ebe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33881 } } }
I20260926 13:15:05.191423 7997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:05.191398 7996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "732c2667b42d4077a702d43bc1f69ebe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "732c2667b42d4077a702d43bc1f69ebe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33881 } } }
I20260926 13:15:05.191584 7996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:05.191854 8006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:15:05.193297 8006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:15:05.194938 8006 catalog_manager.cc:1347] Generated new cluster ID: afc538a455bb43cbb67da8d76f267613
I20260926 13:15:05.194990 8006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:05.204195 8006 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:15:05.204720 8006 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:15:05.206453 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.191402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.0.98.62:33881)
0926 13:15:05.192016 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.192027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.192069 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:15:05.192233 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.192241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.192311 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.192487 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.193268 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.195060 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.196143 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.196885 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.197011 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.200091 (+ 3080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.200096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.200097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.200112 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.202289 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.202784 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.202788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.202788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.202940 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.203189 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.203189 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.203190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.203514 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.203666 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.203835 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.203996 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":76,"threads_started":1}
I20260926 13:15:05.209265 8006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Generated new TSK 0
I20260926 13:15:05.209489 8006 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 13:15:05.241021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33881
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:05.324771 8018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:05.325031 8018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:05.325076 8018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:05.326342 8018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:05.326406 8018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:05.326442 8018 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:15:05.326480 8018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:15:05.328002 8018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:05.328279 8018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:05.328523 8018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:05.333803 8023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:05.333932 8026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:05.333810 8024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.334208 8018 server_base.cc:1034] running on GCE node
I20260926 13:15:05.334439 8018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:05.334683 8018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:05.335852 8018 hybrid_clock.cc:648] HybridClock initialized: now 1790428505335839 us; error 32 us; skew 500 ppm
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:05.338699 8018 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:15:05.339252 8018 webserver.cc:460] Webserver started at http://127.0.98.1:34611/ using document root <none> and password file <none>
I20260926 13:15:05.339507 8018 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:05.339568 8018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:05.339701 8018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:05.341244 8018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "de820df356c507c231955e58e825277f"
server_key_iv: "78f6ab74065e162c784a8e757ee61c91"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.341585 8018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "de820df356c507c231955e58e825277f"
server_key_iv: "78f6ab74065e162c784a8e757ee61c91"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.343116 8018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 13:15:05.343901 8033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.344091 8018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:05.344226 8018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "de820df356c507c231955e58e825277f"
server_key_iv: "78f6ab74065e162c784a8e757ee61c91"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.344314 8018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:05.354067 8018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:05.356166 8018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:05.356352 8018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:05.356894 8018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:05.356966 8018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.357018 8018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:05.357046 8018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.364508 8018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46563
I20260926 13:15:05.364599 8148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46563 every 8 connection(s)
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:05.365036 8018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:15:05.366503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8018
I20260926 13:15:05.366626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:15:05.366875 392 external_mini_cluster.cc:1428] Setting key f4a827d97cef2de81bbf7472c20f0d55
I20260926 13:15:05.368809 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.357505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58449 (local address 127.0.98.62:33881)
0926 13:15:05.357549 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.357551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.357813 (+ 262us) server_negotiation.cc:407] Connection header received
0926 13:15:05.358456 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.358458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.358475 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.358526 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.359440 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.360409 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.362046 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.362451 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.362546 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.365458 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.365460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.365461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.365469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.367262 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.367775 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.367776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.367776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.367830 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.368129 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.368130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.368130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.368252 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.368307 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.368677 (+ 370us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.368746 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:15:05.369179 8042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.357492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.98.1:58449)
0926 13:15:05.357684 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.357720 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.358335 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.358611 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.358614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.359024 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:05.359316 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.359322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.360514 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.360515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.361937 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.361940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.362548 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.362549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.362663 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.363260 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:05.363272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:05.365334 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:05.367333 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.367339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.367342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.367686 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.367896 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.367898 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.367898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.368054 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.368323 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:05.368329 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:05.368522 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.368651 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.368862 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":98,"threads_started":1}
I20260926 13:15:05.369805 8149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33881
I20260926 13:15:05.369925 8149 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:05.370122 8149 heartbeater.cc:511] Master 127.0.98.62:33881 requested a full tablet report, sending...
I20260926 13:15:05.370777 7945 ts_manager.cc:194] Registered new tserver with Master: c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563)
I20260926 13:15:05.371493 7945 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58449
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 13:15:05.380792 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.374594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39521 (local address 127.0.98.62:33881)
0926 13:15:05.374665 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.374667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.374705 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:15:05.374785 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.374786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.374794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.374855 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.375250 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.375795 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.376554 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.376778 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.376847 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.377893 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.377895 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.377895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.377901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.379631 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.379995 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.379999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.379999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.380057 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.380259 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.380260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.380261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.380384 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.380458 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.380622 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.380698 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:15:05.380818 8156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.374462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.98.1:39521)
0926 13:15:05.374630 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.374654 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.374720 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.374860 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.374861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.375088 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:05.375166 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.375168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.375915 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.375916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.376426 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.376428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.376867 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.376868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.376925 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.377435 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:05.377441 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:05.377781 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:05.379713 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.379714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.379714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.379899 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.380126 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.380127 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.380127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.380191 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.380461 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:05.380462 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:05.380508 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.380586 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.380694 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 13:15:05.395040 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33881
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:05.477531 8158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:05.477803 8158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:05.477846 8158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:05.479086 8158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:05.479162 8158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:05.479195 8158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:15:05.479233 8158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:15:05.480763 8158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:05.481060 8158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:05.481315 8158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:05.486610 8164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:05.486626 8163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.486713 8158 server_base.cc:1034] running on GCE node
W20260926 13:15:05.486626 8166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.487061 8158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:05.487317 8158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:05.488459 8158 hybrid_clock.cc:648] HybridClock initialized: now 1790428505488438 us; error 39 us; skew 500 ppm
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:05.491114 8158 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:15:05.491598 8158 webserver.cc:460] Webserver started at http://127.0.98.2:40993/ using document root <none> and password file <none>
I20260926 13:15:05.491899 8158 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:05.491966 8158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:05.492097 8158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:05.493191 8158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "c795ad412d394ebab39fd47c5cda0a5a"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "b0e19a195f038a269bdf69257542f9bc"
server_key_iv: "4055cab2d1880a301df243a651c9dd9f"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.493584 8158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c795ad412d394ebab39fd47c5cda0a5a"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "b0e19a195f038a269bdf69257542f9bc"
server_key_iv: "4055cab2d1880a301df243a651c9dd9f"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.495023 8158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:15:05.495733 8173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.495924 8158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.496021 8158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "c795ad412d394ebab39fd47c5cda0a5a"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "b0e19a195f038a269bdf69257542f9bc"
server_key_iv: "4055cab2d1880a301df243a651c9dd9f"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.496106 8158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:05.517318 8158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:05.519972 8158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:05.520153 8158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:05.520711 8158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:05.520772 8158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.520829 8158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:05.520864 8158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.527983 8158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44117
I20260926 13:15:05.528066 8288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44117 every 8 connection(s)
I20260926 13:15:05.528375 8158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:05.530045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8158
I20260926 13:15:05.530154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:15:05.530382 392 external_mini_cluster.cc:1428] Setting key 9acbb0337529a00cb1f5430f5f68d396
I20260926 13:15:05.532212 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.521383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52733 (local address 127.0.98.62:33881)
0926 13:15:05.521432 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.521434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.521746 (+ 312us) server_negotiation.cc:407] Connection header received
0926 13:15:05.522432 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.522434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.522448 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.522498 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.523416 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.524233 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.526043 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.526310 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.526373 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.529023 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.529025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.529025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.529032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.530730 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.531188 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.531189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.531189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.531255 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.531545 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.531546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.531546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.531667 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.531737 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.532032 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.532122 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:15:05.532415 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.521398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.98.2:52733)
0926 13:15:05.521569 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.521649 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.522312 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.522577 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.522581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.522987 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:05.523306 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.523331 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.524358 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.524359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.525932 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.525936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.526406 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.526408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.526586 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.527263 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:05.527273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:05.528924 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:05.530856 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.530864 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.530867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.531099 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.531338 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.531339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.531339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.531455 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.531738 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:05.531745 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:05.531896 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.531999 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.532168 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260926 13:15:05.533049 8289 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33881
I20260926 13:15:05.533157 8289 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:05.533366 8289 heartbeater.cc:511] Master 127.0.98.62:33881 requested a full tablet report, sending...
I20260926 13:15:05.533756 7946 ts_manager.cc:194] Registered new tserver with Master: c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117)
I20260926 13:15:05.534241 7946 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52733
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:15:05.540084 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.534369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54463 (local address 127.0.98.62:33881)
0926 13:15:05.534410 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.534412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.534483 (+ 71us) server_negotiation.cc:407] Connection header received
0926 13:15:05.534559 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.534560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.534571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.534617 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.534995 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.535471 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.536162 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.536366 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.536438 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.537442 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.537443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.537443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.537450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.538886 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.539236 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.539237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.539237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.539295 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.539496 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.539497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.539498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.539676 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.539754 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.539911 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.539989 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:15:05.540102 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.534251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.98.2:54463)
0926 13:15:05.534417 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.534430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.534500 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.534662 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.534663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.534851 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:05.534905 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.534906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.535559 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.535560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.536070 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.536072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.536437 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.536438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.536509 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.537056 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:05.537060 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:05.537343 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:05.538970 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.538971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.538971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.539150 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.539359 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.539359 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.539360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.539420 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.539746 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:05.539747 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:05.539799 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.539887 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.539990 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 13:15:05.558946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33881
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:05.647378 8298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:05.647658 8298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:05.647699 8298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:05.648949 8298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:05.648995 8298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:05.649016 8298 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 13:15:05.649034 8298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:15:05.650617 8298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33881
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:05.650883 8298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:05.651136 8298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:05.656545 8304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:05.656560 8303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.656564 8298 server_base.cc:1034] running on GCE node
W20260926 13:15:05.656548 8306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:05.656966 8298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:05.657251 8298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:05.658438 8298 hybrid_clock.cc:648] HybridClock initialized: now 1790428505658410 us; error 36 us; skew 500 ppm
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:05.660917 8298 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:15:05.661356 8298 webserver.cc:460] Webserver started at http://127.0.98.3:46865/ using document root <none> and password file <none>
I20260926 13:15:05.661581 8298 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:05.661659 8298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:05.661808 8298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:05.662994 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "a884891081364cc4860aa74b6212fe15"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "3e7f0bc707214ec73d7ebab248b45224"
server_key_iv: "08998a49b23c13d313caeab0c159fbda"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.663374 8298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a884891081364cc4860aa74b6212fe15"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "3e7f0bc707214ec73d7ebab248b45224"
server_key_iv: "08998a49b23c13d313caeab0c159fbda"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.665059 8298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:15:05.665927 8313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.666131 8298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:05.666294 8298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "a884891081364cc4860aa74b6212fe15"
format_stamp: "Formatted at 2026-09-26 13:15:05 on dist-test-slave-f864"
server_key: "3e7f0bc707214ec73d7ebab248b45224"
server_key_iv: "08998a49b23c13d313caeab0c159fbda"
server_key_version: "encryptionkey@0"
I20260926 13:15:05.666379 8298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:05.683470 8298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:05.685410 8298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:05.685577 8298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:05.686118 8298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:05.686252 8298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.686302 8298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:05.686326 8298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:05.693797 8298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43815
I20260926 13:15:05.693929 8428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43815 every 8 connection(s)
I20260926 13:15:05.694231 8298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:15:05.694802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8298
I20260926 13:15:05.694929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:15:05.695143 392 external_mini_cluster.cc:1428] Setting key 145521ed2d0b64ed17549098629e780e
I20260926 13:15:05.697243 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.686657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43157 (local address 127.0.98.62:33881)
0926 13:15:05.686743 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.686746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.686971 (+ 225us) server_negotiation.cc:407] Connection header received
0926 13:15:05.687759 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.687760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.687775 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.687842 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.688859 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.689700 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.691501 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.691742 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.691828 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.694382 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.694383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.694384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.694390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.696158 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.696516 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.696517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.696517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.696567 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.696735 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.696735 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.696736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.696835 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.696885 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.697120 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.697182 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 13:15:05.697540 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.686636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.98.3:43157)
0926 13:15:05.686799 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.686856 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.687621 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.687936 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.687940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.688394 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:05.688735 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.688742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.689814 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.689815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.691402 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.691406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.691856 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.691857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.692034 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.692634 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:05.692644 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:05.694243 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:05.696227 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.696233 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.696234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.696454 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.696607 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.696608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.696608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.696689 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.696893 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:05.696898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:05.697013 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.697098 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.697223 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260926 13:15:05.697997 8429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33881
I20260926 13:15:05.698078 8429 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:05.698293 8429 heartbeater.cc:511] Master 127.0.98.62:33881 requested a full tablet report, sending...
I20260926 13:15:05.698681 7946 ts_manager.cc:194] Registered new tserver with Master: a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815)
I20260926 13:15:05.699148 7946 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43157
I20260926 13:15:05.704676 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.699434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43457 (local address 127.0.98.62:33881)
0926 13:15:05.699489 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.699491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.699603 (+ 112us) server_negotiation.cc:407] Connection header received
0926 13:15:05.699631 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.699632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.699644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.699711 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.700009 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.700493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.701128 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.701308 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.701356 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.702155 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.702156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.702156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.702196 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.703624 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.703923 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.703924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.703924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.703972 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.704198 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.704200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.704200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.704366 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.704440 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.704554 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.704612 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:15:05.704695 8435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.699407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.98.3:43457)
0926 13:15:05.699532 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.699543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.699599 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.699713 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.699713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.699863 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:05.699916 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.699916 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.700573 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.700574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.701063 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.701064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.701368 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.701369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.701416 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.701809 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:05.701813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:05.702081 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:05.703699 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.703700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.703700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.703861 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.704059 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:05.704060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:05.704060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:05.704126 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:05.704437 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:05.704438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:05.704477 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.704532 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.704620 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":77,"threads_started":1}
I20260926 13:15:05.707001 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:15:05.729568 8001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:05.724059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48620 (local address 127.0.98.62:33881)
0926 13:15:05.724130 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.724132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.724155 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:15:05.724190 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.724191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.724200 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.724263 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.724847 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.725338 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.726117 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.726395 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.726478 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.729465 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48620: BlockingRecv error: recv got EOF from 127.0.0.1:48620 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 13:15:05 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:05.736884 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.730390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48624 (local address 127.0.98.62:33881)
0926 13:15:05.730452 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.730454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.730543 (+ 89us) server_negotiation.cc:407] Connection header received
0926 13:15:05.730582 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.730583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.730591 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.730638 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:05.731059 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.731529 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.732188 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.732385 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.732431 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.734609 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:05.734611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:05.734611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:05.734617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:05.736021 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.736310 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.736311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.736311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.736360 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:05.736498 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:05.736498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:05.736498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:05.736588 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:05.736655 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.736737 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.736794 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:15:05.738498 7946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:15:05.740047 7946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:15:05.746426 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.742393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.0.98.3:43815)
0926 13:15:05.742563 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.742569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.742680 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:15:05.742817 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.742819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.742836 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.742898 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:05.743122 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.743880 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.745227 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.745973 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.746043 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.746121 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.746270 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.746325 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":68,"threads_started":1}
I20260926 13:15:05.746430 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.742319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43815 (local address 127.0.0.1:43092)
0926 13:15:05.742614 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.742626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.742709 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.742990 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.742991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.742993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:05.743045 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.743047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.743970 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.743971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.745136 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.745139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.746061 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.746062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.746138 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.746154 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.746253 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.746309 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":18,"thread_start_us":84,"threads_started":1}
I20260926 13:15:05.746615 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.742405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.0.98.2:44117)
0926 13:15:05.742586 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.742591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.742602 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:15:05.742667 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.742669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.742683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.742757 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:05.742891 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.744063 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.745260 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.745972 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.746042 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.746120 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.746468 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.746510 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260926 13:15:05.746433 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.742314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44117 (local address 127.0.0.1:38822)
0926 13:15:05.742469 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.742509 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.742573 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.742768 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.742770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.742773 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:05.742834 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.742836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.744159 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.744160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.745148 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.745157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.746156 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.746156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.746256 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.746260 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.746311 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.746346 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":55,"threads_started":1}
I20260926 13:15:05.747061 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.742711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.0.98.1:46563)
0926 13:15:05.743021 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.743025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.743034 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:15:05.743068 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.743070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.743086 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.743152 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:05.743328 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.744390 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.745510 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.745992 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.746078 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.746156 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.746475 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.746953 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":74,"threads_started":1}
I20260926 13:15:05.747077 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.742506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.0.1:46130)
0926 13:15:05.742868 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.742878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.742917 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.743209 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.743210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.743212 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:05.743240 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.743241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.744471 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.744472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.745428 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.745430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.746218 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.746219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.746281 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.746294 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.746339 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.746375 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":62,"threads_started":1}
I20260926 13:15:05.747431 8363 tablet_service.cc:1467] Processing CreateTablet for tablet ad2b6ce5e2824e8bb2127c15b827a82a (DEFAULT_TABLE table=test-table [id=0c4bf288689941a1965c2375dc3839a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:05.747476 8223 tablet_service.cc:1467] Processing CreateTablet for tablet ad2b6ce5e2824e8bb2127c15b827a82a (DEFAULT_TABLE table=test-table [id=0c4bf288689941a1965c2375dc3839a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:05.747738 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2b6ce5e2824e8bb2127c15b827a82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.747740 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2b6ce5e2824e8bb2127c15b827a82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.750891 8458 tablet_bootstrap.cc:492] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Bootstrap starting.
I20260926 13:15:05.751245 8083 tablet_service.cc:1467] Processing CreateTablet for tablet ad2b6ce5e2824e8bb2127c15b827a82a (DEFAULT_TABLE table=test-table [id=0c4bf288689941a1965c2375dc3839a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:05.751504 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad2b6ce5e2824e8bb2127c15b827a82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.751531 8458 tablet_bootstrap.cc:654] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.751760 8458 log.cc:824] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:05.752422 8458 tablet_bootstrap.cc:492] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: No bootstrap required, opened a new log
I20260926 13:15:05.752467 8458 ts_tablet_manager.cc:1397] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:05.753728 8457 tablet_bootstrap.cc:492] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Bootstrap starting.
I20260926 13:15:05.754065 8458 raft_consensus.cc:348] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.754273 8458 raft_consensus.cc:374] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.754326 8458 raft_consensus.cc:729] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c795ad412d394ebab39fd47c5cda0a5a, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.754426 8457 tablet_bootstrap.cc:654] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.754503 8458 consensus_queue.cc:260] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.754805 8458 ts_tablet_manager.cc:1428] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:15:05.754838 8289 heartbeater.cc:503] Master 127.0.98.62:33881 was elected leader, sending a full tablet report...
I20260926 13:15:05.754947 8457 log.cc:824] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:05.755252 8460 tablet_bootstrap.cc:492] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Bootstrap starting.
I20260926 13:15:05.755891 8457 tablet_bootstrap.cc:492] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: No bootstrap required, opened a new log
I20260926 13:15:05.755885 8460 tablet_bootstrap.cc:654] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.755961 8457 ts_tablet_manager.cc:1397] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:05.756225 8460 log.cc:824] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:05.756994 8460 tablet_bootstrap.cc:492] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: No bootstrap required, opened a new log
I20260926 13:15:05.757067 8460 ts_tablet_manager.cc:1397] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:05.758116 8457 raft_consensus.cc:348] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.758361 8457 raft_consensus.cc:374] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.758404 8457 raft_consensus.cc:729] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a884891081364cc4860aa74b6212fe15, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.758528 8457 consensus_queue.cc:260] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.758805 8457 ts_tablet_manager.cc:1428] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:15:05.758848 8429 heartbeater.cc:503] Master 127.0.98.62:33881 was elected leader, sending a full tablet report...
I20260926 13:15:05.759164 8460 raft_consensus.cc:348] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.759320 8460 raft_consensus.cc:374] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.759344 8460 raft_consensus.cc:729] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332267ba5014da3aeb8b84e12fa5ba8, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.759482 8460 consensus_queue.cc:260] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.759768 8460 ts_tablet_manager.cc:1428] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:15:05.759788 8149 heartbeater.cc:503] Master 127.0.98.62:33881 was elected leader, sending a full tablet report...
W20260926 13:15:05.779007 8290 tablet.cc:2369] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:15:05.865828 8150 tablet.cc:2369] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:15:05.943526 8464 raft_consensus.cc:482] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:15:05.943647 8464 raft_consensus.cc:504] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.943995 8464 leader_election.cc:288] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117), c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563)
W20260926 13:15:05.944826 8430 tablet.cc:2369] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:15:05.947134 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.944234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50915 (local address 127.0.98.2:44117)
0926 13:15:05.944276 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.944278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.944426 (+ 148us) server_negotiation.cc:407] Connection header received
0926 13:15:05.944457 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.944457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.944467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.944555 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:05.944768 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.945333 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.946383 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.946768 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.946853 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.946863 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.947019 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.947055 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 13:15:05.947149 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.944330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50305 (local address 127.0.98.1:46563)
0926 13:15:05.944400 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.944402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.944504 (+ 102us) server_negotiation.cc:407] Connection header received
0926 13:15:05.944531 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.944532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.944543 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.944646 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:05.944861 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.945354 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.946397 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.946770 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.946864 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.946873 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.947040 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.947078 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:15:05.947170 8322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.944190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44117 (local address 127.0.98.3:50915)
0926 13:15:05.944271 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.944307 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.944369 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.944574 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.944574 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.944576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:05.944688 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.944690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.945426 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.945427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.946291 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.946293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.946851 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.946852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.946946 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.946949 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.947015 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.947058 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":52}
I20260926 13:15:05.947513 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b6ce5e2824e8bb2127c15b827a82a" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a" is_pre_election: true
I20260926 13:15:05.947678 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.944207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.98.3:50305)
0926 13:15:05.944428 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:05.944437 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:05.944493 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:05.944669 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:05.944670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:05.944671 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:05.944723 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.944725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.945439 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.945440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.946301 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:05.946303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.946868 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.946869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:05.946963 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.946967 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:15:05.947034 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:15:05.947074 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":1,"thread_start_us":137,"threads_started":1}
I20260926 13:15:05.947671 8243 raft_consensus.cc:2454] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 0.
I20260926 13:15:05.947979 8317 leader_election.cc:302] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:05.947999 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b6ce5e2824e8bb2127c15b827a82a" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" is_pre_election: true
I20260926 13:15:05.948138 8103 raft_consensus.cc:2454] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 0.
I20260926 13:15:05.948140 8464 raft_consensus.cc:2781] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:15:05.948206 8464 raft_consensus.cc:482] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:15:05.948244 8464 raft_consensus.cc:3037] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:05.949311 8464 raft_consensus.cc:504] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.949465 8464 leader_election.cc:288] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 election: Requested vote from peers c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117), c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563)
I20260926 13:15:05.949600 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b6ce5e2824e8bb2127c15b827a82a" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
I20260926 13:15:05.949620 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad2b6ce5e2824e8bb2127c15b827a82a" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a"
I20260926 13:15:05.949685 8243 raft_consensus.cc:3037] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:05.949687 8103 raft_consensus.cc:3037] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:05.950578 8103 raft_consensus.cc:2454] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 1.
I20260926 13:15:05.950578 8243 raft_consensus.cc:2454] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 1.
I20260926 13:15:05.950742 8317 leader_election.cc:302] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:05.950841 8464 raft_consensus.cc:2781] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:05.950984 8464 raft_consensus.cc:686] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 1 LEADER]: Becoming Leader. State: Replica: a884891081364cc4860aa74b6212fe15, State: Running, Role: LEADER
I20260926 13:15:05.951082 8464 consensus_queue.cc:237] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:05.951905 7946 catalog_manager.cc:5495] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 reported cstate change: term changed from 0 to 1, leader changed from <none> to a884891081364cc4860aa74b6212fe15 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } health_report { overall_health: UNKNOWN } } }
I20260926 13:15:05.976989 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:05.974129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.0.98.3:43815)
0926 13:15:05.974187 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:05.974189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:05.974216 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:15:05.974286 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:05.974287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:05.974304 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:05.974368 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:05.974540 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.975003 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.976230 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:05.976416 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:05.976466 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:05.976665 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:05.976758 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:05.976820 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:05.976888 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:15:05.976906 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:15:05.978787 8243 raft_consensus.cc:1261] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Refusing update from remote peer a884891081364cc4860aa74b6212fe15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:05.978818 8103 raft_consensus.cc:1261] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Refusing update from remote peer a884891081364cc4860aa74b6212fe15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:05.979103 8467 consensus_queue.cc:1035] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:05.979242 8464 consensus_queue.cc:1035] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:05.981411 8470 mvcc.cc:205] Tried to move back new op lower bound from 7333595160486207488 to 7333595160376426496. Current Snapshot: MvccSnapshot[applied={T|T < 7333595160486207488}]
I20260926 13:15:05.982077 8471 mvcc.cc:205] Tried to move back new op lower bound from 7333595160486207488 to 7333595160376426496. Current Snapshot: MvccSnapshot[applied={T|T < 7333595160486207488}]
I20260926 13:15:05.982120 8472 mvcc.cc:205] Tried to move back new op lower bound from 7333595160486207488 to 7333595160376426496. Current Snapshot: MvccSnapshot[applied={T|T < 7333595160486207488}]
I20260926 13:15:05.984416 7946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:05.984498 7946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:05.985136 7946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Sending DeleteTablet for 3 replicas of tablet ad2b6ce5e2824e8bb2127c15b827a82a
I20260926 13:15:05.985507 8363 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2b6ce5e2824e8bb2127c15b827a82a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43092
I20260926 13:15:05.985652 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2b6ce5e2824e8bb2127c15b827a82a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260926 13:15:05.985653 7946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:15:05.985783 7946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:15:05.985912 8482 tablet_replica.cc:330] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: stopping tablet replica
I20260926 13:15:05.985893 8083 tablet_service.cc:1514] Processing DeleteTablet for tablet ad2b6ce5e2824e8bb2127c15b827a82a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46130
I20260926 13:15:05.986006 8482 raft_consensus.cc:2229] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:05.986114 8482 raft_consensus.cc:2258] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:05.987097 8223 tablet_service.cc:1467] Processing CreateTablet for tablet 61faa4cc09424f80a3dbdc15dcb26d6b (DEFAULT_TABLE table=test-table [id=1a82792c6918467895af7a250a528173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:05.987098 8363 tablet_service.cc:1467] Processing CreateTablet for tablet 61faa4cc09424f80a3dbdc15dcb26d6b (DEFAULT_TABLE table=test-table [id=1a82792c6918467895af7a250a528173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:05.987221 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61faa4cc09424f80a3dbdc15dcb26d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.987221 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61faa4cc09424f80a3dbdc15dcb26d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.988559 8457 tablet_bootstrap.cc:492] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Bootstrap starting.
I20260926 13:15:05.988698 8083 tablet_service.cc:1467] Processing CreateTablet for tablet 61faa4cc09424f80a3dbdc15dcb26d6b (DEFAULT_TABLE table=test-table [id=1a82792c6918467895af7a250a528173]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:05.988835 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61faa4cc09424f80a3dbdc15dcb26d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:05.989013 8457 tablet_bootstrap.cc:654] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.989356 8458 tablet_bootstrap.cc:492] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Bootstrap starting.
I20260926 13:15:05.989765 8458 tablet_bootstrap.cc:654] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.990374 8457 tablet_bootstrap.cc:492] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: No bootstrap required, opened a new log
I20260926 13:15:05.990383 8458 tablet_bootstrap.cc:492] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: No bootstrap required, opened a new log
I20260926 13:15:05.990424 8458 ts_tablet_manager.cc:1397] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:05.990424 8457 ts_tablet_manager.cc:1397] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:05.990442 8460 tablet_bootstrap.cc:492] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Bootstrap starting.
I20260926 13:15:05.990572 8457 raft_consensus.cc:348] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:05.990572 8458 raft_consensus.cc:348] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:05.990635 8458 raft_consensus.cc:374] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.990635 8457 raft_consensus.cc:374] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.990657 8457 raft_consensus.cc:729] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a884891081364cc4860aa74b6212fe15, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.990657 8458 raft_consensus.cc:729] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c795ad412d394ebab39fd47c5cda0a5a, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.990698 8458 consensus_queue.cc:260] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:05.990698 8457 consensus_queue.cc:260] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:05.990795 8457 ts_tablet_manager.cc:1428] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:05.990803 8458 ts_tablet_manager.cc:1428] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:05.990844 8460 tablet_bootstrap.cc:654] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:05.991459 8482 ts_tablet_manager.cc:1905] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:05.991910 8460 tablet_bootstrap.cc:492] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: No bootstrap required, opened a new log
I20260926 13:15:05.991964 8460 ts_tablet_manager.cc:1397] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:05.992147 8460 raft_consensus.cc:348] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:05.992223 8460 raft_consensus.cc:374] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:05.992249 8460 raft_consensus.cc:729] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332267ba5014da3aeb8b84e12fa5ba8, State: Initialized, Role: FOLLOWER
I20260926 13:15:05.992309 8460 consensus_queue.cc:260] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:05.992458 8460 ts_tablet_manager.cc:1428] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:05.993285 8482 ts_tablet_manager.cc:1918] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:05.993391 8482 log.cc:1196] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/ad2b6ce5e2824e8bb2127c15b827a82a
I20260926 13:15:05.993631 8482 ts_tablet_manager.cc:1939] T ad2b6ce5e2824e8bb2127c15b827a82a P c795ad412d394ebab39fd47c5cda0a5a: Deleting consensus metadata
I20260926 13:15:05.994021 7933 catalog_manager.cc:4841] TS c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117): tablet ad2b6ce5e2824e8bb2127c15b827a82a (table test-table [id=0c4bf288689941a1965c2375dc3839a3]) successfully deleted
I20260926 13:15:05.994261 8483 tablet_replica.cc:330] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: stopping tablet replica
I20260926 13:15:05.994415 8483 raft_consensus.cc:2229] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:05.994537 8483 raft_consensus.cc:2258] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:05.994957 8483 ts_tablet_manager.cc:1905] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:05.997015 8483 ts_tablet_manager.cc:1918] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:05.997133 8483 log.cc:1196] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/ad2b6ce5e2824e8bb2127c15b827a82a
I20260926 13:15:05.997406 8483 ts_tablet_manager.cc:1939] T ad2b6ce5e2824e8bb2127c15b827a82a P c332267ba5014da3aeb8b84e12fa5ba8: Deleting consensus metadata
I20260926 13:15:05.997841 7932 catalog_manager.cc:4841] TS c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563): tablet ad2b6ce5e2824e8bb2127c15b827a82a (table test-table [id=0c4bf288689941a1965c2375dc3839a3]) successfully deleted
I20260926 13:15:06.002264 8481 tablet_replica.cc:330] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: stopping tablet replica
I20260926 13:15:06.002395 8481 raft_consensus.cc:2229] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:15:06.002517 8481 raft_consensus.cc:2258] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.003036 8481 ts_tablet_manager.cc:1905] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.004616 8481 ts_tablet_manager.cc:1918] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:06.004714 8481 log.cc:1196] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/ad2b6ce5e2824e8bb2127c15b827a82a
I20260926 13:15:06.004980 8481 ts_tablet_manager.cc:1939] T ad2b6ce5e2824e8bb2127c15b827a82a P a884891081364cc4860aa74b6212fe15: Deleting consensus metadata
I20260926 13:15:06.005385 7932 catalog_manager.cc:4841] TS a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815): tablet ad2b6ce5e2824e8bb2127c15b827a82a (table test-table [id=0c4bf288689941a1965c2375dc3839a3]) successfully deleted
I20260926 13:15:06.183737 8477 raft_consensus.cc:482] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:15:06.183835 8477 raft_consensus.cc:504] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.184000 8477 leader_election.cc:288] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563), c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117)
I20260926 13:15:06.184273 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61faa4cc09424f80a3dbdc15dcb26d6b" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a" is_pre_election: true
I20260926 13:15:06.184286 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61faa4cc09424f80a3dbdc15dcb26d6b" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" is_pre_election: true
I20260926 13:15:06.184382 8103 raft_consensus.cc:2454] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 0.
I20260926 13:15:06.184382 8243 raft_consensus.cc:2454] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 0.
I20260926 13:15:06.184556 8316 leader_election.cc:302] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c332267ba5014da3aeb8b84e12fa5ba8; no voters:
I20260926 13:15:06.184674 8477 raft_consensus.cc:2781] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:15:06.184731 8477 raft_consensus.cc:482] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:15:06.184753 8477 raft_consensus.cc:3037] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.185469 8477 raft_consensus.cc:504] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.185617 8477 leader_election.cc:288] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 election: Requested vote from peers c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563), c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117)
I20260926 13:15:06.185751 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61faa4cc09424f80a3dbdc15dcb26d6b" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a"
I20260926 13:15:06.185845 8243 raft_consensus.cc:3037] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.185858 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61faa4cc09424f80a3dbdc15dcb26d6b" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
I20260926 13:15:06.185976 8103 raft_consensus.cc:3037] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.186556 8243 raft_consensus.cc:2454] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 1.
I20260926 13:15:06.186770 8317 leader_election.cc:302] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:06.186826 8103 raft_consensus.cc:2454] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 1.
I20260926 13:15:06.186861 8477 raft_consensus.cc:2781] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:06.186909 8477 raft_consensus.cc:686] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 1 LEADER]: Becoming Leader. State: Replica: a884891081364cc4860aa74b6212fe15, State: Running, Role: LEADER
I20260926 13:15:06.186960 8477 consensus_queue.cc:237] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.187502 7945 catalog_manager.cc:5495] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 reported cstate change: term changed from 0 to 1, leader changed from <none> to a884891081364cc4860aa74b6212fe15 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } health_report { overall_health: HEALTHY } } }
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428505, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:06.241863 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.235114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.0.98.62:33881)
0926 13:15:06.235165 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.235167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.235300 (+ 133us) server_negotiation.cc:407] Connection header received
0926 13:15:06.235373 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.235374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.235384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.235441 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:06.236015 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.236613 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.237326 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.237519 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.237590 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.239760 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:06.239761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:06.239762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:06.239769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:06.240918 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:06.241260 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:06.241261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:06.241261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:06.241308 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:06.241472 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:06.241473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:06.241473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:06.241566 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:06.241614 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.241728 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.241796 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:15:06.241935 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.235054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33881 (local address 127.0.0.1:48640)
0926 13:15:06.235210 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.235221 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.235318 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.235447 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.235447 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.235841 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:06.235900 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.235902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.236698 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.236699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.237229 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.237231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.237587 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.237587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.237633 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.238283 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:06.238289 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:06.239679 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:06.240989 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:06.240996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:06.240997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:06.241186 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:06.241365 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:06.241366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:06.241366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:06.241418 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:06.241617 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:06.241623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:06.241663 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.241713 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.241845 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":51,"thread_start_us":96,"threads_started":1}
I20260926 13:15:06.243834 7945 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48640:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 13:15:06.244055 7945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:15:06.245373 8223 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9ef427290f4d1da656d68ee7216528 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9f4a777f26545f1a23bd8935f60fe74]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:15:06.245414 8363 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9ef427290f4d1da656d68ee7216528 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9f4a777f26545f1a23bd8935f60fe74]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:15:06.245419 8083 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9ef427290f4d1da656d68ee7216528 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b9f4a777f26545f1a23bd8935f60fe74]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 13:15:06.245533 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9ef427290f4d1da656d68ee7216528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.245530 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9ef427290f4d1da656d68ee7216528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.245530 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9ef427290f4d1da656d68ee7216528. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.246979 8458 tablet_bootstrap.cc:492] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: Bootstrap starting.
I20260926 13:15:06.246979 8460 tablet_bootstrap.cc:492] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: Bootstrap starting.
I20260926 13:15:06.247006 8457 tablet_bootstrap.cc:492] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: Bootstrap starting.
I20260926 13:15:06.247408 8460 tablet_bootstrap.cc:654] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.247408 8458 tablet_bootstrap.cc:654] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.247411 8457 tablet_bootstrap.cc:654] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.248030 8457 tablet_bootstrap.cc:492] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: No bootstrap required, opened a new log
I20260926 13:15:06.248059 8460 tablet_bootstrap.cc:492] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: No bootstrap required, opened a new log
I20260926 13:15:06.248080 8457 ts_tablet_manager.cc:1397] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:06.248106 8460 ts_tablet_manager.cc:1397] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:06.248204 8457 raft_consensus.cc:348] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.248281 8457 raft_consensus.cc:374] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.248304 8457 raft_consensus.cc:729] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a884891081364cc4860aa74b6212fe15, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.248266 8460 raft_consensus.cc:348] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.248332 8460 raft_consensus.cc:374] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.248364 8458 tablet_bootstrap.cc:492] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: No bootstrap required, opened a new log
I20260926 13:15:06.248354 8457 consensus_queue.cc:260] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.248383 8460 raft_consensus.cc:729] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332267ba5014da3aeb8b84e12fa5ba8, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.248404 8458 ts_tablet_manager.cc:1397] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:06.248425 8460 consensus_queue.cc:260] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.248452 8457 ts_tablet_manager.cc:1428] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:06.248508 8460 ts_tablet_manager.cc:1428] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:06.248469 8477 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.248557 8477 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.248546 8458 raft_consensus.cc:348] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.248607 8458 raft_consensus.cc:374] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.248631 8458 raft_consensus.cc:729] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c795ad412d394ebab39fd47c5cda0a5a, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.248610 8465 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.248670 8458 consensus_queue.cc:260] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.248696 8465 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.248764 8458 ts_tablet_manager.cc:1428] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:06.248843 8461 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.248930 8461 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.383294 8477 raft_consensus.cc:482] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:15:06.383456 8477 raft_consensus.cc:504] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.383694 8477 leader_election.cc:288] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117), c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563)
I20260926 13:15:06.383963 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9ef427290f4d1da656d68ee7216528" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" is_pre_election: true
I20260926 13:15:06.383965 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9ef427290f4d1da656d68ee7216528" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a" is_pre_election: true
I20260926 13:15:06.384080 8103 raft_consensus.cc:2454] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 0.
I20260926 13:15:06.384080 8243 raft_consensus.cc:2454] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 0.
I20260926 13:15:06.384265 8316 leader_election.cc:302] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c332267ba5014da3aeb8b84e12fa5ba8; no voters:
I20260926 13:15:06.384413 8477 raft_consensus.cc:2781] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:15:06.384472 8477 raft_consensus.cc:482] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:15:06.384490 8477 raft_consensus.cc:3037] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.385187 8477 raft_consensus.cc:504] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.385349 8477 leader_election.cc:288] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 election: Requested vote from peers c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117), c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563)
I20260926 13:15:06.385546 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9ef427290f4d1da656d68ee7216528" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a"
I20260926 13:15:06.385564 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9ef427290f4d1da656d68ee7216528" candidate_uuid: "a884891081364cc4860aa74b6212fe15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
I20260926 13:15:06.385638 8103 raft_consensus.cc:3037] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.385643 8243 raft_consensus.cc:3037] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.386329 8103 raft_consensus.cc:2454] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 1.
I20260926 13:15:06.386515 8316 leader_election.cc:302] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c332267ba5014da3aeb8b84e12fa5ba8; no voters:
I20260926 13:15:06.386611 8243 raft_consensus.cc:2454] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a884891081364cc4860aa74b6212fe15 in term 1.
I20260926 13:15:06.386659 8477 raft_consensus.cc:2781] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:06.386745 8477 raft_consensus.cc:686] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [term 1 LEADER]: Becoming Leader. State: Replica: a884891081364cc4860aa74b6212fe15, State: Running, Role: LEADER
I20260926 13:15:06.386837 8477 consensus_queue.cc:237] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } }
I20260926 13:15:06.387115 8464 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: TxnStatusTablet state changed. Reason: New leader a884891081364cc4860aa74b6212fe15. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.387179 8464 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:15:06.387342 8498 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 13:15:06.387504 8498 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 13:15:06.387542 7945 catalog_manager.cc:5495] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 reported cstate change: term changed from 0 to 1, leader changed from <none> to a884891081364cc4860aa74b6212fe15 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } health_report { overall_health: UNKNOWN } } }
I20260926 13:15:06.434353 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.431397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.98.3:43815)
0926 13:15:06.431462 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.431464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.431616 (+ 152us) server_negotiation.cc:407] Connection header received
0926 13:15:06.431705 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.431706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.431715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.431763 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:06.431897 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.432371 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.433537 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.433724 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.433778 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.434005 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.434041 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.434093 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.434237 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.434275 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:15:06.434361 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.431450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43815 (local address 127.0.0.1:56522)
0926 13:15:06.431540 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.431553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.431622 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.431779 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.431780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.431781 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:15:06.431837 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.431839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.432457 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.432458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.433436 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.433439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.433806 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.433807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.433879 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.433899 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.434117 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.434134 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.434213 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.434270 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 13:15:06.435614 8103 raft_consensus.cc:1261] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Refusing update from remote peer a884891081364cc4860aa74b6212fe15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.435632 8243 raft_consensus.cc:1261] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Refusing update from remote peer a884891081364cc4860aa74b6212fe15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.435912 8477 consensus_queue.cc:1035] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.436015 8464 consensus_queue.cc:1035] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.436523 8465 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: TxnStatusTablet state changed. Reason: New leader a884891081364cc4860aa74b6212fe15. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.436548 8461 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: TxnStatusTablet state changed. Reason: New leader a884891081364cc4860aa74b6212fe15. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.436610 8465 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.436635 8461 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.437217 8476 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.437315 8476 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:15:06.437438 8465 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.437513 8465 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P c332267ba5014da3aeb8b84e12fa5ba8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.437755 8477 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.437832 8477 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P a884891081364cc4860aa74b6212fe15: This TxnStatusTablet replica's current role is: LEADER
I20260926 13:15:06.438043 8461 tablet_replica.cc:439] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a884891081364cc4860aa74b6212fe15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } } peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } }
I20260926 13:15:06.438124 8461 tablet_replica.cc:442] T 7e9ef427290f4d1da656d68ee7216528 P c795ad412d394ebab39fd47c5cda0a5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 13:15:06.444131 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.441671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42637 (local address 127.0.98.3:43815)
0926 13:15:06.441719 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.441721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.441845 (+ 124us) server_negotiation.cc:407] Connection header received
0926 13:15:06.441916 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.441917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.441925 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.441968 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:06.442104 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.442631 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.443411 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.443606 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.443690 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.443863 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.443899 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.443957 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.444008 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.444050 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":26}
I20260926 13:15:06.444140 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.441609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43815 (local address 127.0.98.3:42637)
0926 13:15:06.441767 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.441777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.441858 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.441968 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.441969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.441970 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:15:06.442036 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.442037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.442730 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.442731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.443325 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.443327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.443695 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.443696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.443754 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.443776 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.443957 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.443959 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.444006 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.444068 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260926 13:15:06.446532 8243 raft_consensus.cc:1261] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Refusing update from remote peer a884891081364cc4860aa74b6212fe15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.446540 8103 raft_consensus.cc:1261] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Refusing update from remote peer a884891081364cc4860aa74b6212fe15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.446724 8477 consensus_queue.cc:1035] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.446843 8476 consensus_queue.cc:1035] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.458225 7944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:06.458359 7944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:06.458930 7944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Sending DeleteTablet for 3 replicas of tablet 61faa4cc09424f80a3dbdc15dcb26d6b
I20260926 13:15:06.459215 8083 tablet_service.cc:1514] Processing DeleteTablet for tablet 61faa4cc09424f80a3dbdc15dcb26d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46130
I20260926 13:15:06.459213 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet 61faa4cc09424f80a3dbdc15dcb26d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260926 13:15:06.459372 8483 tablet_replica.cc:330] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: stopping tablet replica
I20260926 13:15:06.459441 8483 raft_consensus.cc:2229] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:06.459488 8483 raft_consensus.cc:2258] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.459560 8482 tablet_replica.cc:330] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: stopping tablet replica
I20260926 13:15:06.459626 8482 raft_consensus.cc:2229] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:06.459668 8482 raft_consensus.cc:2258] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.459975 8483 ts_tablet_manager.cc:1905] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.461926 8483 ts_tablet_manager.cc:1918] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:15:06.461997 8483 log.cc:1196] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/61faa4cc09424f80a3dbdc15dcb26d6b
I20260926 13:15:06.462028 8482 ts_tablet_manager.cc:1905] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.462299 8483 ts_tablet_manager.cc:1939] T 61faa4cc09424f80a3dbdc15dcb26d6b P c332267ba5014da3aeb8b84e12fa5ba8: Deleting consensus metadata
I20260926 13:15:06.462925 7932 catalog_manager.cc:4841] TS c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563): tablet 61faa4cc09424f80a3dbdc15dcb26d6b (table test-table [id=1a82792c6918467895af7a250a528173]) successfully deleted
I20260926 13:15:06.463153 8363 tablet_service.cc:1514] Processing DeleteTablet for tablet 61faa4cc09424f80a3dbdc15dcb26d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43092
I20260926 13:15:06.463254 8481 tablet_replica.cc:330] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: stopping tablet replica
I20260926 13:15:06.463315 8481 raft_consensus.cc:2229] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:15:06.463379 8481 raft_consensus.cc:2258] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.463778 8481 ts_tablet_manager.cc:1905] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.463795 8482 ts_tablet_manager.cc:1918] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:15:06.463843 8482 log.cc:1196] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/61faa4cc09424f80a3dbdc15dcb26d6b
I20260926 13:15:06.464051 8482 ts_tablet_manager.cc:1939] T 61faa4cc09424f80a3dbdc15dcb26d6b P c795ad412d394ebab39fd47c5cda0a5a: Deleting consensus metadata
I20260926 13:15:06.464519 7933 catalog_manager.cc:4841] TS c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117): tablet 61faa4cc09424f80a3dbdc15dcb26d6b (table test-table [id=1a82792c6918467895af7a250a528173]) successfully deleted
I20260926 13:15:06.465252 8481 ts_tablet_manager.cc:1918] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:15:06.465312 8481 log.cc:1196] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/61faa4cc09424f80a3dbdc15dcb26d6b
I20260926 13:15:06.465529 8481 ts_tablet_manager.cc:1939] T 61faa4cc09424f80a3dbdc15dcb26d6b P a884891081364cc4860aa74b6212fe15: Deleting consensus metadata
I20260926 13:15:06.465934 7932 catalog_manager.cc:4841] TS a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815): tablet 61faa4cc09424f80a3dbdc15dcb26d6b (table test-table [id=1a82792c6918467895af7a250a528173]) successfully deleted
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[7901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:15:06.485282 8001 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:06.479314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.0.98.62:33881)
0926 13:15:06.479400 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.479403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.479529 (+ 126us) server_negotiation.cc:407] Connection header received
0926 13:15:06.479614 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.479615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.479628 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.479714 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:06.480313 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.480817 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.481635 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.481837 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.481912 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.485205 (+ 3293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37712: BlockingRecv error: recv got EOF from 127.0.0.1:37712 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[7901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428506, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 13:15:06.493417 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.486129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.0.98.62:33881)
0926 13:15:06.486216 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.486218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.486329 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:15:06.486423 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.486423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.486433 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.486490 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:06.486937 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.487405 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.488088 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.488277 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.488350 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.490708 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:06.490710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:06.490710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:06.490717 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:06.492280 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:06.492630 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:06.492631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:06.492631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:06.492691 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:06.492882 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:06.492883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:06.492884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:06.492992 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:06.493091 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.493210 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.493293 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:15:06.494915 7944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:15:06.495049 7944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:15:06.496412 8363 tablet_service.cc:1467] Processing CreateTablet for tablet daf7665ac0d543af99332ab5acaba906 (DEFAULT_TABLE table=test-table [id=add97584d52940b9bf3f8db7a805f045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:06.496450 8223 tablet_service.cc:1467] Processing CreateTablet for tablet daf7665ac0d543af99332ab5acaba906 (DEFAULT_TABLE table=test-table [id=add97584d52940b9bf3f8db7a805f045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:06.496563 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf7665ac0d543af99332ab5acaba906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.496563 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf7665ac0d543af99332ab5acaba906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.497891 8458 tablet_bootstrap.cc:492] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Bootstrap starting.
I20260926 13:15:06.497928 8083 tablet_service.cc:1467] Processing CreateTablet for tablet daf7665ac0d543af99332ab5acaba906 (DEFAULT_TABLE table=test-table [id=add97584d52940b9bf3f8db7a805f045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:06.498071 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf7665ac0d543af99332ab5acaba906. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.498365 8458 tablet_bootstrap.cc:654] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.499050 8458 tablet_bootstrap.cc:492] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: No bootstrap required, opened a new log
I20260926 13:15:06.499104 8458 ts_tablet_manager.cc:1397] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:06.499275 8458 raft_consensus.cc:348] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.499351 8458 raft_consensus.cc:374] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.499369 8458 raft_consensus.cc:729] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c795ad412d394ebab39fd47c5cda0a5a, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.499411 8458 consensus_queue.cc:260] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.499449 8457 tablet_bootstrap.cc:492] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Bootstrap starting.
I20260926 13:15:06.499497 8458 ts_tablet_manager.cc:1428] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:06.499818 8457 tablet_bootstrap.cc:654] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.500325 8460 tablet_bootstrap.cc:492] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Bootstrap starting.
I20260926 13:15:06.500689 8460 tablet_bootstrap.cc:654] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.501271 8457 tablet_bootstrap.cc:492] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: No bootstrap required, opened a new log
I20260926 13:15:06.501334 8457 ts_tablet_manager.cc:1397] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:06.501488 8457 raft_consensus.cc:348] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.501564 8457 raft_consensus.cc:374] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.501591 8457 raft_consensus.cc:729] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a884891081364cc4860aa74b6212fe15, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.501636 8457 consensus_queue.cc:260] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.501726 8457 ts_tablet_manager.cc:1428] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:06.501920 8460 tablet_bootstrap.cc:492] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: No bootstrap required, opened a new log
I20260926 13:15:06.501964 8460 ts_tablet_manager.cc:1397] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:06.502106 8460 raft_consensus.cc:348] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.502213 8460 raft_consensus.cc:374] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.502240 8460 raft_consensus.cc:729] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332267ba5014da3aeb8b84e12fa5ba8, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.502290 8460 consensus_queue.cc:260] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.502388 8460 ts_tablet_manager.cc:1428] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:06.523813 8461 raft_consensus.cc:482] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:15:06.523918 8461 raft_consensus.cc:504] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.524302 8461 leader_election.cc:288] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563), a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815)
I20260926 13:15:06.527357 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.524607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36683 (local address 127.0.98.1:46563)
0926 13:15:06.524812 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.524814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.524822 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:06.524848 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.524848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.524859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.524927 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:06.525088 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.525610 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.526628 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.527002 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.527088 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.527100 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.527235 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.527281 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":67,"threads_started":1}
I20260926 13:15:06.527357 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.524483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.98.2:36683)
0926 13:15:06.524618 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.524631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.524687 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.524931 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.524932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.524934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:06.525011 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.525014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.525700 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.525701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.526543 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.526545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.527088 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.527089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.527160 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.527164 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.527230 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.527270 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260926 13:15:06.527396 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.524798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52875 (local address 127.0.98.3:43815)
0926 13:15:06.524875 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.524877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.524961 (+ 84us) server_negotiation.cc:407] Connection header received
0926 13:15:06.525024 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.525025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.525035 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.525089 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:06.525264 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.525716 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.526699 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.527049 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.527130 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.527144 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.527294 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.527330 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:15:06.527477 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.524702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43815 (local address 127.0.98.2:52875)
0926 13:15:06.524886 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.524897 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.524961 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.525101 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.525101 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.525103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:06.525188 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.525189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.525813 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.525814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.526610 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.526612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.527147 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.527148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.527218 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.527221 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.527288 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.527326 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":36,"threads_started":1}
I20260926 13:15:06.527697 8383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf7665ac0d543af99332ab5acaba906" candidate_uuid: "c795ad412d394ebab39fd47c5cda0a5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a884891081364cc4860aa74b6212fe15" is_pre_election: true
I20260926 13:15:06.527765 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf7665ac0d543af99332ab5acaba906" candidate_uuid: "c795ad412d394ebab39fd47c5cda0a5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" is_pre_election: true
I20260926 13:15:06.527856 8383 raft_consensus.cc:2454] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c795ad412d394ebab39fd47c5cda0a5a in term 0.
I20260926 13:15:06.527863 8103 raft_consensus.cc:2454] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c795ad412d394ebab39fd47c5cda0a5a in term 0.
I20260926 13:15:06.528275 8176 leader_election.cc:302] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a884891081364cc4860aa74b6212fe15, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:06.528439 8461 raft_consensus.cc:2781] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:15:06.528491 8461 raft_consensus.cc:482] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:15:06.528509 8461 raft_consensus.cc:3037] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.529340 8461 raft_consensus.cc:504] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.529474 8461 leader_election.cc:288] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [CANDIDATE]: Term 1 election: Requested vote from peers c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563), a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815)
I20260926 13:15:06.529635 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf7665ac0d543af99332ab5acaba906" candidate_uuid: "c795ad412d394ebab39fd47c5cda0a5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c332267ba5014da3aeb8b84e12fa5ba8"
I20260926 13:15:06.529666 8383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf7665ac0d543af99332ab5acaba906" candidate_uuid: "c795ad412d394ebab39fd47c5cda0a5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a884891081364cc4860aa74b6212fe15"
I20260926 13:15:06.529733 8383 raft_consensus.cc:3037] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.529735 8103 raft_consensus.cc:3037] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.530436 8103 raft_consensus.cc:2454] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c795ad412d394ebab39fd47c5cda0a5a in term 1.
I20260926 13:15:06.530525 8383 raft_consensus.cc:2454] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c795ad412d394ebab39fd47c5cda0a5a in term 1.
I20260926 13:15:06.530620 8176 leader_election.cc:302] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c332267ba5014da3aeb8b84e12fa5ba8, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:06.530743 8461 raft_consensus.cc:2781] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:06.530853 8461 raft_consensus.cc:686] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 1 LEADER]: Becoming Leader. State: Replica: c795ad412d394ebab39fd47c5cda0a5a, State: Running, Role: LEADER
I20260926 13:15:06.530930 8461 consensus_queue.cc:237] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.531690 7944 catalog_manager.cc:5495] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a reported cstate change: term changed from 0 to 1, leader changed from <none> to c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c795ad412d394ebab39fd47c5cda0a5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } health_report { overall_health: UNKNOWN } } }
I20260926 13:15:06.536769 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.534245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.0.98.2:44117)
0926 13:15:06.534401 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.534403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.534411 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:06.534450 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.534451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.534467 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.534536 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:06.534647 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.535091 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.535995 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.536192 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.536255 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.536417 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.536512 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.536579 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.536688 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.536705 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260926 13:15:06.537884 8103 raft_consensus.cc:1261] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Refusing update from remote peer c795ad412d394ebab39fd47c5cda0a5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.537925 8383 raft_consensus.cc:1261] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Refusing update from remote peer c795ad412d394ebab39fd47c5cda0a5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.538241 8527 consensus_queue.cc:1035] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.538372 8461 consensus_queue.cc:1035] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.541877 7944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:06.541972 7944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:06.543292 7944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Sending DeleteTablet for 3 replicas of tablet daf7665ac0d543af99332ab5acaba906
I20260926 13:15:06.543570 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet daf7665ac0d543af99332ab5acaba906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260926 13:15:06.543687 8482 tablet_replica.cc:330] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: stopping tablet replica
I20260926 13:15:06.543740 8482 raft_consensus.cc:2229] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:15:06.543684 8083 tablet_service.cc:1514] Processing DeleteTablet for tablet daf7665ac0d543af99332ab5acaba906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46130
I20260926 13:15:06.543830 8482 raft_consensus.cc:2258] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.543913 8483 tablet_replica.cc:330] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: stopping tablet replica
I20260926 13:15:06.543965 8483 raft_consensus.cc:2229] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:06.543838 8363 tablet_service.cc:1514] Processing DeleteTablet for tablet daf7665ac0d543af99332ab5acaba906 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43092
I20260926 13:15:06.544013 8483 raft_consensus.cc:2258] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.544051 8481 tablet_replica.cc:330] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: stopping tablet replica
I20260926 13:15:06.544106 8481 raft_consensus.cc:2229] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:06.544225 8481 raft_consensus.cc:2258] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.544404 7944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:15:06.544548 7944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:15:06.544638 8483 ts_tablet_manager.cc:1905] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.544858 8481 ts_tablet_manager.cc:1905] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.546358 8483 ts_tablet_manager.cc:1918] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:06.546409 8483 log.cc:1196] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/daf7665ac0d543af99332ab5acaba906
I20260926 13:15:06.546442 8482 ts_tablet_manager.cc:1905] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.546653 8483 ts_tablet_manager.cc:1939] T daf7665ac0d543af99332ab5acaba906 P c332267ba5014da3aeb8b84e12fa5ba8: Deleting consensus metadata
I20260926 13:15:06.547644 8481 ts_tablet_manager.cc:1918] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:06.547727 8481 log.cc:1196] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/daf7665ac0d543af99332ab5acaba906
I20260926 13:15:06.547962 8481 ts_tablet_manager.cc:1939] T daf7665ac0d543af99332ab5acaba906 P a884891081364cc4860aa74b6212fe15: Deleting consensus metadata
I20260926 13:15:06.548116 8482 ts_tablet_manager.cc:1918] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:06.548177 8482 log.cc:1196] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/daf7665ac0d543af99332ab5acaba906
I20260926 13:15:06.548395 8482 ts_tablet_manager.cc:1939] T daf7665ac0d543af99332ab5acaba906 P c795ad412d394ebab39fd47c5cda0a5a: Deleting consensus metadata
I20260926 13:15:06.548368 7932 catalog_manager.cc:4841] TS c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563): tablet daf7665ac0d543af99332ab5acaba906 (table test-table [id=add97584d52940b9bf3f8db7a805f045]) successfully deleted
I20260926 13:15:06.549011 7933 catalog_manager.cc:4841] TS c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117): tablet daf7665ac0d543af99332ab5acaba906 (table test-table [id=add97584d52940b9bf3f8db7a805f045]) successfully deleted
I20260926 13:15:06.549137 7932 catalog_manager.cc:4841] TS a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815): tablet daf7665ac0d543af99332ab5acaba906 (table test-table [id=add97584d52940b9bf3f8db7a805f045]) successfully deleted
I20260926 13:15:06.550696 8363 tablet_service.cc:1467] Processing CreateTablet for tablet e4da98e0fc664d3199ee890c7652932c (DEFAULT_TABLE table=test-table [id=8e818cf1a18d4bb0a8170f9877d93685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:06.550696 8083 tablet_service.cc:1467] Processing CreateTablet for tablet e4da98e0fc664d3199ee890c7652932c (DEFAULT_TABLE table=test-table [id=8e818cf1a18d4bb0a8170f9877d93685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:06.550806 8083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4da98e0fc664d3199ee890c7652932c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.550806 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4da98e0fc664d3199ee890c7652932c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.551743 8223 tablet_service.cc:1467] Processing CreateTablet for tablet e4da98e0fc664d3199ee890c7652932c (DEFAULT_TABLE table=test-table [id=8e818cf1a18d4bb0a8170f9877d93685]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:06.551847 8223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4da98e0fc664d3199ee890c7652932c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:06.552016 8457 tablet_bootstrap.cc:492] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Bootstrap starting.
I20260926 13:15:06.552325 8457 tablet_bootstrap.cc:654] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.552830 8457 tablet_bootstrap.cc:492] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: No bootstrap required, opened a new log
I20260926 13:15:06.552860 8457 ts_tablet_manager.cc:1397] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:06.552945 8457 raft_consensus.cc:348] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.552994 8457 raft_consensus.cc:374] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.553006 8457 raft_consensus.cc:729] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a884891081364cc4860aa74b6212fe15, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.553045 8460 tablet_bootstrap.cc:492] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Bootstrap starting.
I20260926 13:15:06.553071 8457 consensus_queue.cc:260] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.553176 8457 ts_tablet_manager.cc:1428] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:06.553663 8460 tablet_bootstrap.cc:654] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.554097 8458 tablet_bootstrap.cc:492] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Bootstrap starting.
I20260926 13:15:06.554498 8458 tablet_bootstrap.cc:654] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:06.554937 8460 tablet_bootstrap.cc:492] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: No bootstrap required, opened a new log
I20260926 13:15:06.554987 8460 ts_tablet_manager.cc:1397] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:15:06.555140 8458 tablet_bootstrap.cc:492] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: No bootstrap required, opened a new log
I20260926 13:15:06.555176 8458 ts_tablet_manager.cc:1397] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:06.555132 8460 raft_consensus.cc:348] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.555208 8460 raft_consensus.cc:374] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.555231 8460 raft_consensus.cc:729] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c332267ba5014da3aeb8b84e12fa5ba8, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.555272 8460 consensus_queue.cc:260] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.555294 8458 raft_consensus.cc:348] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.555341 8458 raft_consensus.cc:374] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:06.555353 8458 raft_consensus.cc:729] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c795ad412d394ebab39fd47c5cda0a5a, State: Initialized, Role: FOLLOWER
I20260926 13:15:06.555361 8460 ts_tablet_manager.cc:1428] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:06.555413 8458 consensus_queue.cc:260] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.555505 8458 ts_tablet_manager.cc:1428] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:06.644171 8465 raft_consensus.cc:482] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:15:06.644290 8465 raft_consensus.cc:504] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.644651 8465 leader_election.cc:288] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117), a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815)
I20260926 13:15:06.648375 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.644842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44117 (local address 127.0.98.1:43809)
0926 13:15:06.644983 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.644996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.645055 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.645251 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.645252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.645254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:06.645354 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.645358 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.646372 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.646373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.647334 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.647337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.648097 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.648098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.648174 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.648179 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.648235 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.648274 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":40,"thread_start_us":76,"threads_started":1}
I20260926 13:15:06.648398 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.644957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43809 (local address 127.0.98.2:44117)
0926 13:15:06.645092 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.645094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.645102 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:06.645141 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.645142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.645151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.645223 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:06.645441 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.646254 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.647427 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.647990 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.648064 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.648076 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.648283 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.648327 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 13:15:06.648671 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.645012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39913 (local address 127.0.98.3:43815)
0926 13:15:06.645057 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.645059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.645116 (+ 57us) server_negotiation.cc:407] Connection header received
0926 13:15:06.645227 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.645228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.645249 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.645302 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:06.645521 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.646250 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.647678 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.648265 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.648331 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.648341 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.648543 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.648600 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:15:06.648799 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4da98e0fc664d3199ee890c7652932c" candidate_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a" is_pre_election: true
I20260926 13:15:06.648885 8243 raft_consensus.cc:2454] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c332267ba5014da3aeb8b84e12fa5ba8 in term 0.
I20260926 13:15:06.648913 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.644842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43815 (local address 127.0.98.1:39913)
0926 13:15:06.645058 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.645067 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.645140 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.645312 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.645313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.645315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:06.645396 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.645398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.646367 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.646367 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.647542 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.647544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.648372 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.648373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.648451 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.648456 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.648785 (+ 329us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.648832 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260926 13:15:06.649127 8383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4da98e0fc664d3199ee890c7652932c" candidate_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a884891081364cc4860aa74b6212fe15" is_pre_election: true
I20260926 13:15:06.649148 8037 leader_election.cc:302] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c332267ba5014da3aeb8b84e12fa5ba8, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:06.649219 8383 raft_consensus.cc:2454] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c332267ba5014da3aeb8b84e12fa5ba8 in term 0.
I20260926 13:15:06.649266 8465 raft_consensus.cc:2781] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:15:06.649305 8465 raft_consensus.cc:482] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:15:06.649338 8465 raft_consensus.cc:3037] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.650110 8465 raft_consensus.cc:504] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.650292 8465 leader_election.cc:288] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [CANDIDATE]: Term 1 election: Requested vote from peers c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117), a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815)
I20260926 13:15:06.650454 8243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4da98e0fc664d3199ee890c7652932c" candidate_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c795ad412d394ebab39fd47c5cda0a5a"
I20260926 13:15:06.650527 8243 raft_consensus.cc:3037] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.650484 8383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4da98e0fc664d3199ee890c7652932c" candidate_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a884891081364cc4860aa74b6212fe15"
I20260926 13:15:06.650565 8383 raft_consensus.cc:3037] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:06.651130 8243 raft_consensus.cc:2454] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c332267ba5014da3aeb8b84e12fa5ba8 in term 1.
I20260926 13:15:06.651314 8037 leader_election.cc:302] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c332267ba5014da3aeb8b84e12fa5ba8, c795ad412d394ebab39fd47c5cda0a5a; no voters:
I20260926 13:15:06.651382 8383 raft_consensus.cc:2454] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c332267ba5014da3aeb8b84e12fa5ba8 in term 1.
I20260926 13:15:06.651414 8465 raft_consensus.cc:2781] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:06.651564 8465 raft_consensus.cc:686] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 LEADER]: Becoming Leader. State: Replica: c332267ba5014da3aeb8b84e12fa5ba8, State: Running, Role: LEADER
I20260926 13:15:06.651650 8465 consensus_queue.cc:237] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } }
I20260926 13:15:06.652565 7945 catalog_manager.cc:5495] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c332267ba5014da3aeb8b84e12fa5ba8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 } health_report { overall_health: UNKNOWN } } }
I20260926 13:15:06.671842 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.669569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.0.98.1:46563)
0926 13:15:06.669623 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.669625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.669678 (+ 53us) server_negotiation.cc:407] Connection header received
0926 13:15:06.669766 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.669767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.669793 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.669855 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:06.669939 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.670539 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.671225 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.671408 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.671462 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.671571 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.671670 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.671719 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.671766 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.671784 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:15:06.676505 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.674359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35047 (local address 127.0.98.3:43815)
0926 13:15:06.674425 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.674427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.674520 (+ 93us) server_negotiation.cc:407] Connection header received
0926 13:15:06.674570 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.674571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.674579 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.674640 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:06.674756 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.675207 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.675917 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.676114 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.676170 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.676309 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.676335 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.676372 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.676423 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.676441 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 13:15:06.676533 8543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.674291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43815 (local address 127.0.98.1:35047)
0926 13:15:06.674461 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.674479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.674528 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.674643 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.674644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.674645 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:15:06.674707 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.674708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.675269 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.675269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.675829 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.675831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.676181 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.676181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.676225 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.676238 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.676385 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.676387 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.676421 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.676450 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":107,"threads_started":1}
I20260926 13:15:06.678579 8383 raft_consensus.cc:1261] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Refusing update from remote peer c332267ba5014da3aeb8b84e12fa5ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.678615 8243 raft_consensus.cc:1261] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Refusing update from remote peer c332267ba5014da3aeb8b84e12fa5ba8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:06.678881 8465 consensus_queue.cc:1035] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c795ad412d394ebab39fd47c5cda0a5a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.679009 8539 consensus_queue.cc:1035] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a884891081364cc4860aa74b6212fe15" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:06.685588 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.683365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33119 (local address 127.0.98.1:46563)
0926 13:15:06.683400 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:06.683402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:06.683410 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:06.683443 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:06.683444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:06.683453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:06.683511 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:06.683638 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.684125 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.684768 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.684949 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.684997 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.685309 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.685346 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.685422 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:06.685490 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:15:06.685510 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:15:06.685607 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.683318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.98.3:33119)
0926 13:15:06.683357 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.683368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:06.683399 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:06.683526 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:06.683526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:06.683528 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 13:15:06.683566 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.683567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.684176 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.684177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.684702 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:06.684703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:06.685045 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:06.685046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:06.685093 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:06.685101 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 13:15:06.685426 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 13:15:06.685428 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:06.685486 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:15:06.685536 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 13:15:06.692137 7944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:06.692230 7944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:06.692860 7944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 732c2667b42d4077a702d43bc1f69ebe: Sending DeleteTablet for 3 replicas of tablet e4da98e0fc664d3199ee890c7652932c
I20260926 13:15:06.693115 8083 tablet_service.cc:1514] Processing DeleteTablet for tablet e4da98e0fc664d3199ee890c7652932c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46130
I20260926 13:15:06.693120 8363 tablet_service.cc:1514] Processing DeleteTablet for tablet e4da98e0fc664d3199ee890c7652932c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43092
I20260926 13:15:06.693221 8481 tablet_replica.cc:330] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: stopping tablet replica
I20260926 13:15:06.693230 8223 tablet_service.cc:1514] Processing DeleteTablet for tablet e4da98e0fc664d3199ee890c7652932c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:06 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38822
I20260926 13:15:06.693285 8481 raft_consensus.cc:2229] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:06.693320 8481 raft_consensus.cc:2258] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.693329 8483 tablet_replica.cc:330] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: stopping tablet replica
I20260926 13:15:06.693387 8483 raft_consensus.cc:2229] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:15:06.693472 8483 raft_consensus.cc:2258] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.693544 8482 tablet_replica.cc:330] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: stopping tablet replica
I20260926 13:15:06.693593 8482 raft_consensus.cc:2229] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:06.693624 8482 raft_consensus.cc:2258] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:06.693652 8481 ts_tablet_manager.cc:1905] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.693998 8482 ts_tablet_manager.cc:1905] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.695474 8482 ts_tablet_manager.cc:1918] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:15:06.695567 8482 log.cc:1196] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/e4da98e0fc664d3199ee890c7652932c
I20260926 13:15:06.695778 8482 ts_tablet_manager.cc:1939] T e4da98e0fc664d3199ee890c7652932c P c795ad412d394ebab39fd47c5cda0a5a: Deleting consensus metadata
I20260926 13:15:06.696126 7933 catalog_manager.cc:4841] TS c795ad412d394ebab39fd47c5cda0a5a (127.0.98.2:44117): tablet e4da98e0fc664d3199ee890c7652932c (table test-table [id=8e818cf1a18d4bb0a8170f9877d93685]) successfully deleted
I20260926 13:15:06.696306 8483 ts_tablet_manager.cc:1905] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:06.697547 8483 ts_tablet_manager.cc:1918] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:15:06.697605 8483 log.cc:1196] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/e4da98e0fc664d3199ee890c7652932c
I20260926 13:15:06.697830 8483 ts_tablet_manager.cc:1939] T e4da98e0fc664d3199ee890c7652932c P c332267ba5014da3aeb8b84e12fa5ba8: Deleting consensus metadata
I20260926 13:15:06.697868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8018
I20260926 13:15:06.697877 8481 ts_tablet_manager.cc:1918] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 13:15:06.697933 8481 log.cc:1196] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/e4da98e0fc664d3199ee890c7652932c
I20260926 13:15:06.698122 8481 ts_tablet_manager.cc:1939] T e4da98e0fc664d3199ee890c7652932c P a884891081364cc4860aa74b6212fe15: Deleting consensus metadata
I20260926 13:15:06.699841 7932 catalog_manager.cc:4841] TS a884891081364cc4860aa74b6212fe15 (127.0.98.3:43815): tablet e4da98e0fc664d3199ee890c7652932c (table test-table [id=8e818cf1a18d4bb0a8170f9877d93685]) successfully deleted
W20260926 13:15:06.703701 7932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46563 (error 108)
I20260926 13:15:06.704000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8158
I20260926 13:15:06.704743 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:06.704086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46563 (local address 127.0.0.1:44142)
0926 13:15:06.704641 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:06.704686 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":40,"threads_started":1}
W20260926 13:15:06.705189 7932 catalog_manager.cc:4584] TS c332267ba5014da3aeb8b84e12fa5ba8 (127.0.98.1:46563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4da98e0fc664d3199ee890c7652932c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46563: connect: Connection refused (error 111)
I20260926 13:15:06.710064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8298
I20260926 13:15:06.716464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 7914
2026-09-26T13:15:06Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2814 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[8552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[8552](info): set up 2 sockets
Sep 26 13:15:06 dist-test-slave-f864 krb5kdc[8552](info): commencing operation
krb5kdc: starting...
W20260926 13:15:07.754854 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:15:07 dist-test-slave-f864 krb5kdc[8552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:15:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:15:07.875140 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46693
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:07.954927 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:07.955127 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:07.955148 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:07.956300 8565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:07.956341 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:07.956354 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:07.956367 8565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:07.956379 8565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:07.957851 8565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:07.958088 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:07.958352 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:07.963181 8573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:07.963202 8570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:07.963199 8571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:07.963456 8565 server_base.cc:1034] running on GCE node
I20260926 13:15:07.963760 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:07.964090 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:07.965268 8565 hybrid_clock.cc:648] HybridClock initialized: now 1790428507965269 us; error 52 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T13:15:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1255 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:15:08 dist-test-slave-f864 krb5kdc[8578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:15:08 dist-test-slave-f864 krb5kdc[8578](info): set up 2 sockets
Sep 26 13:15:08 dist-test-slave-f864 krb5kdc[8578](info): commencing operation
krb5kdc: starting...
W20260926 13:15:09.030702 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:15:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:15:09.152726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34135
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:09.233603 8591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:09.233807 8591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:09.233832 8591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:09.235034 8591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:09.235077 8591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:09.235092 8591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:09.235105 8591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:09.235116 8591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:09.236557 8591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:09.236744 8591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:09.236914 8591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:09.241804 8597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:09.241801 8596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:09.241882 8591 server_base.cc:1034] running on GCE node
W20260926 13:15:09.241801 8599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:09.242296 8591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:09.242549 8591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:09.243696 8591 hybrid_clock.cc:648] HybridClock initialized: now 1790428509243686 us; error 29 us; skew 500 ppm
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:09.255548 8591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:15:09.256003 8591 webserver.cc:460] Webserver started at http://127.0.98.62:35571/ using document root <none> and password file <none>
I20260926 13:15:09.256278 8591 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:09.256350 8591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:09.256501 8591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:09.257591 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "d34aa3ff7353410abc65aaae470dcca7"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "3eb07f88528a1b00662b6f32e6abcb5d"
server_key_iv: "c85c9f9f64c337c054084ececb2d63c9"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.257951 8591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "d34aa3ff7353410abc65aaae470dcca7"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "3eb07f88528a1b00662b6f32e6abcb5d"
server_key_iv: "c85c9f9f64c337c054084ececb2d63c9"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.259409 8591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:15:09.260251 8606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.260470 8591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:09.260576 8591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "d34aa3ff7353410abc65aaae470dcca7"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "3eb07f88528a1b00662b6f32e6abcb5d"
server_key_iv: "c85c9f9f64c337c054084ececb2d63c9"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.260660 8591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:09.268296 8591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:09.270429 8591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:09.270606 8591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:09.274439 8591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34135
I20260926 13:15:09.274487 8658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34135 every 8 connection(s)
I20260926 13:15:09.275018 8591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:15:09.275739 8659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:09.277727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8591
I20260926 13:15:09.277837 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:15:09.278000 392 external_mini_cluster.cc:1428] Setting key 149a55a278a0312a4c014518cc81e177
I20260926 13:15:09.277985 8659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Bootstrap starting.
I20260926 13:15:09.278771 8659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:09.279095 8659 log.cc:824] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:09.279822 8659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: No bootstrap required, opened a new log
I20260926 13:15:09.282246 8659 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:09.282395 8659 raft_consensus.cc:374] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:09.282419 8659 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34aa3ff7353410abc65aaae470dcca7, State: Initialized, Role: FOLLOWER
I20260926 13:15:09.282582 8659 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:09.282656 8659 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:15:09.282698 8659 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:15:09.282786 8659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:09.283504 8659 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:09.283648 8659 leader_election.cc:302] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34aa3ff7353410abc65aaae470dcca7; no voters:
I20260926 13:15:09.283847 8659 leader_election.cc:288] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:15:09.283910 8664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:09.284431 8664 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 LEADER]: Becoming Leader. State: Replica: d34aa3ff7353410abc65aaae470dcca7, State: Running, Role: LEADER
I20260926 13:15:09.284578 8659 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:15:09.284569 8664 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:09.285219 8665 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34aa3ff7353410abc65aaae470dcca7. Latest consensus state: current_term: 1 leader_uuid: "d34aa3ff7353410abc65aaae470dcca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } } }
I20260926 13:15:09.285323 8665 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:09.285480 8666 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d34aa3ff7353410abc65aaae470dcca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } } }
I20260926 13:15:09.285535 8666 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: This master's current role is: LEADER
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:09.285902 8673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:15:09.286432 8673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:15:09.288383 8673 catalog_manager.cc:1347] Generated new cluster ID: ddfad7b53a9049d1b8bc0573c192696d
I20260926 13:15:09.288435 8673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:15:09.290311 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.278581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38342 (local address 127.0.98.62:34135)
0926 13:15:09.278768 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:09.278777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:09.278810 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:15:09.279398 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:09.279406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:09.279454 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:09.279622 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:09.280304 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.281760 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.282729 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.283351 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.283470 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.286429 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:09.286433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:09.286434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:09.286446 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:09.288373 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.288778 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.288781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.288781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.288874 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.289058 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.289059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.289059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.289338 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:09.289471 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:09.289931 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 13:15:09.290079 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260926 13:15:09.308421 8673 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:15:09.309221 8673 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:15:09.320794 8673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Generated new TSK 0
I20260926 13:15:09.321031 8673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:15:09.350277 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34135
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:09.433116 8685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:09.433378 8685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:09.433418 8685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:09.434643 8685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:09.434713 8685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:09.434756 8685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:15:09.436283 8685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:09.436558 8685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:09.436800 8685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:09.441887 8690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:09.441887 8691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:09.441898 8693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:09.442332 8685 server_base.cc:1034] running on GCE node
I20260926 13:15:09.442574 8685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:09.442778 8685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:09.443944 8685 hybrid_clock.cc:648] HybridClock initialized: now 1790428509443931 us; error 26 us; skew 500 ppm
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:09.446786 8685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:15:09.447206 8685 webserver.cc:460] Webserver started at http://127.0.98.1:32849/ using document root <none> and password file <none>
I20260926 13:15:09.447443 8685 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:09.447512 8685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:09.447650 8685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:09.448721 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "6a7c87fc6a1242c98159c552199e9ee5"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "e8974e91ae787479b0c98f391da3cafc"
server_key_iv: "c7c37e3cad71caaf5ef972786d75351c"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.449088 8685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6a7c87fc6a1242c98159c552199e9ee5"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "e8974e91ae787479b0c98f391da3cafc"
server_key_iv: "c7c37e3cad71caaf5ef972786d75351c"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.450536 8685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:15:09.451352 8700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.451527 8685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:09.451627 8685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "6a7c87fc6a1242c98159c552199e9ee5"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "e8974e91ae787479b0c98f391da3cafc"
server_key_iv: "c7c37e3cad71caaf5ef972786d75351c"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.451710 8685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:09.488059 8685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:09.490622 8685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:09.490794 8685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:09.491041 8685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:09.491375 8685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:09.491436 8685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.491472 8685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:09.491500 8685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.497094 8685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41141
I20260926 13:15:09.497136 8813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41141 every 8 connection(s)
I20260926 13:15:09.497488 8685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:09.505967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8685
I20260926 13:15:09.506103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:15:09.506301 392 external_mini_cluster.cc:1428] Setting key c2bd64bb84525e539ae3a5133789e0d6
I20260926 13:15:09.506843 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.498261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36343 (local address 127.0.98.62:34135)
0926 13:15:09.498330 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:09.498333 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:09.498545 (+ 212us) server_negotiation.cc:407] Connection header received
0926 13:15:09.499469 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:09.499470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:09.499480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:09.499528 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:09.500121 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.500605 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.501578 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.501768 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.501815 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.504011 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:09.504013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:09.504013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:09.504020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:09.505401 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.505786 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.505787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.505788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.505854 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.506109 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.506110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.506111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.506286 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:09.506365 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:09.506668 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 13:15:09.506754 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:15:09.507050 8816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.498253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:36343)
0926 13:15:09.498418 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:09.498455 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:09.499384 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:09.499597 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:09.499601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:09.499845 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:09.500037 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:09.500041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.500693 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.500694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:09.501507 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:09.501509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.501836 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.501837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:09.501935 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.502440 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:09.502448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:09.503927 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:09.505473 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:09.505478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:09.505480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:09.505702 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:09.505922 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:09.505923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:09.505923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:09.506022 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:09.506356 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:09.506362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:09.506493 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 13:15:09.506626 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:15:09.506805 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260926 13:15:09.507542 8814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34135
I20260926 13:15:09.507658 8814 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:09.507853 8814 heartbeater.cc:511] Master 127.0.98.62:34135 requested a full tablet report, sending...
I20260926 13:15:09.508430 8623 ts_manager.cc:194] Registered new tserver with Master: 6a7c87fc6a1242c98159c552199e9ee5 (127.0.98.1:41141)
I20260926 13:15:09.509126 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36343
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:15:09.533591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34135
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:09.615726 8819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:09.615940 8819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:09.615965 8819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:09.617079 8819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:09.617120 8819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:09.617144 8819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:15:09.618706 8819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:09.618896 8819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:09.619076 8819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:09.625198 8825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:09.625244 8819 server_base.cc:1034] running on GCE node
W20260926 13:15:09.625200 8827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:09.625259 8824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:09.625779 8819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:09.626039 8819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:09.627239 8819 hybrid_clock.cc:648] HybridClock initialized: now 1790428509627221 us; error 64 us; skew 500 ppm
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:09.630110 8819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:15:09.630661 8819 webserver.cc:460] Webserver started at http://127.0.98.2:46553/ using document root <none> and password file <none>
I20260926 13:15:09.630874 8819 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:09.630916 8819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:09.631011 8819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:09.632104 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "f034c99b632445ed864ad528403c7def"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "52a62213339c1cd565c20ab2876e8b4a"
server_key_iv: "cfe42f60397fa80dc3641b7738a6449a"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.632429 8819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f034c99b632445ed864ad528403c7def"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "52a62213339c1cd565c20ab2876e8b4a"
server_key_iv: "cfe42f60397fa80dc3641b7738a6449a"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.634423 8819 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260926 13:15:09.635380 8834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.635648 8819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:09.635780 8819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "f034c99b632445ed864ad528403c7def"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "52a62213339c1cd565c20ab2876e8b4a"
server_key_iv: "cfe42f60397fa80dc3641b7738a6449a"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.635867 8819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:09.655364 8819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:09.657734 8819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:09.657908 8819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:09.658200 8819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:09.658564 8819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:09.658636 8819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.658680 8819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:09.658706 8819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.664758 8819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46703
I20260926 13:15:09.664803 8947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46703 every 8 connection(s)
I20260926 13:15:09.665158 8819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:15:09.668781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8819
I20260926 13:15:09.668874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:15:09.669062 392 external_mini_cluster.cc:1428] Setting key 788c083919b636ff4fe82098ad44a160
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:09.675906 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.666041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47809 (local address 127.0.98.62:34135)
0926 13:15:09.666135 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:09.666138 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:09.666338 (+ 200us) server_negotiation.cc:407] Connection header received
0926 13:15:09.667363 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:09.667364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:09.667378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:09.667445 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:09.668104 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.668605 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.670114 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.670385 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.670477 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.672953 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:09.672955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:09.672956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:09.672964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:09.674488 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.674915 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.674917 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.674917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.674977 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.675238 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.675239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.675239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.675359 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:09.675439 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:09.675721 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 13:15:09.675807 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 13:15:09.676129 8950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.665953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:47809)
0926 13:15:09.666197 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:09.666247 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:09.667237 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:09.667527 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:09.667531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:09.667796 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:09.667998 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:09.668002 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.668703 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.668704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:09.670019 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:09.670022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.670487 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.670488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:09.670629 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.671195 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:09.671208 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:09.672847 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:09.674586 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:09.674593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:09.674595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:09.674836 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:09.675046 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:09.675047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:09.675047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:09.675169 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:09.675428 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:09.675434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:09.675587 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 13:15:09.675703 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 13:15:09.675871 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260926 13:15:09.676620 8948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34135
I20260926 13:15:09.676715 8948 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:09.676921 8948 heartbeater.cc:511] Master 127.0.98.62:34135 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:15:09.677347 8623 ts_manager.cc:194] Registered new tserver with Master: f034c99b632445ed864ad528403c7def (127.0.98.2:46703)
I20260926 13:15:09.677814 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:15:09.696805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34135
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:09.779078 8953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:09.779302 8953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:09.779326 8953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:09.780536 8953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:09.780586 8953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:09.780613 8953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:15:09.782222 8953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:09.782472 8953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:09.782712 8953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:09.788092 8958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:09.788098 8961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:09.788098 8959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:09.788169 8953 server_base.cc:1034] running on GCE node
I20260926 13:15:09.788511 8953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:09.788802 8953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:09.789971 8953 hybrid_clock.cc:648] HybridClock initialized: now 1790428509789943 us; error 41 us; skew 500 ppm
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:09.792626 8953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:15:09.793018 8953 webserver.cc:460] Webserver started at http://127.0.98.3:46643/ using document root <none> and password file <none>
I20260926 13:15:09.793285 8953 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:09.793330 8953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:09.793457 8953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:09.794631 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "a77c4de56a114d3d9b0645ca80822b32"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "17552a594c15d50994dd34e4eee925ef"
server_key_iv: "5afe59713782d17877d0a9ed8a4e588e"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.794994 8953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a77c4de56a114d3d9b0645ca80822b32"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "17552a594c15d50994dd34e4eee925ef"
server_key_iv: "5afe59713782d17877d0a9ed8a4e588e"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.796356 8953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:09.797071 8968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.797297 8953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:09.797395 8953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "a77c4de56a114d3d9b0645ca80822b32"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "17552a594c15d50994dd34e4eee925ef"
server_key_iv: "5afe59713782d17877d0a9ed8a4e588e"
server_key_version: "encryptionkey@0"
I20260926 13:15:09.797483 8953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:09.820749 8953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:09.823091 8953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:09.823256 8953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:09.823493 8953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:09.823813 8953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:09.823868 8953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.823911 8953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:09.823938 8953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:09.829744 8953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45161
I20260926 13:15:09.829787 9081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45161 every 8 connection(s)
I20260926 13:15:09.830119 8953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:15:09.832000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 8953
I20260926 13:15:09.832105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:15:09.832288 392 external_mini_cluster.cc:1428] Setting key 3d7f0073663fff23bef71ecec4c30fc5
Sep 26 13:15:09 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:09.840291 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.830931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55575 (local address 127.0.98.62:34135)
0926 13:15:09.830990 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:09.830993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:09.831186 (+ 193us) server_negotiation.cc:407] Connection header received
0926 13:15:09.832307 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:09.832309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:09.832319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:09.832413 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:09.833316 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.833816 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.834846 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.835030 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.835084 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.837096 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:09.837098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:09.837098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:09.837104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:09.838811 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.839370 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.839371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.839371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.839445 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:09.839712 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:09.839712 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:09.839713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:09.839824 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:09.839876 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:09.840161 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 13:15:09.840225 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:15:09.840559 9084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:09.830872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:55575)
0926 13:15:09.831043 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:09.831093 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:09.832196 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:09.832488 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:09.832492 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:09.832906 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:09.833230 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:09.833243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.833896 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.833896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:09.834777 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:09.834779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:09.835110 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:09.835111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:09.835210 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:09.835663 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:09.835672 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:09.837016 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:09.838928 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:09.838935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:09.838937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:09.839274 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:09.839551 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:09.839552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:09.839552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:09.839653 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:09.839867 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:09.839870 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:09.840039 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 13:15:09.840132 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:15:09.840256 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260926 13:15:09.841007 9082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34135
I20260926 13:15:09.841114 9082 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:09.841290 9082 heartbeater.cc:511] Master 127.0.98.62:34135 requested a full tablet report, sending...
I20260926 13:15:09.841652 8623 ts_manager.cc:194] Registered new tserver with Master: a77c4de56a114d3d9b0645ca80822b32 (127.0.98.3:45161)
I20260926 13:15:09.842021 8623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55575
I20260926 13:15:09.843920 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 13:15:09.920627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8685
I20260926 13:15:09.925629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8819
I20260926 13:15:09.930244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8953
I20260926 13:15:09.934731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 8591
I20260926 13:15:09.939759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34135
--webserver_interface=127.0.98.62
--webserver_port=35571
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34135
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:10.020838 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:10.021123 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:10.021166 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:10.022406 9091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:10.022490 9091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:10.022516 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:10.022536 9091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:10.022568 9091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:10.024138 9091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34135
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35571
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:10.024449 9091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:10.024699 9091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:10.029834 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.029836 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.029834 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:10.030122 9091 server_base.cc:1034] running on GCE node
I20260926 13:15:10.030404 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:10.030733 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:10.031904 9091 hybrid_clock.cc:648] HybridClock initialized: now 1790428510031888 us; error 31 us; skew 500 ppm
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:10.034659 9091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:15:10.035070 9091 webserver.cc:460] Webserver started at http://127.0.98.62:35571/ using document root <none> and password file <none>
I20260926 13:15:10.035264 9091 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:10.035312 9091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:10.036693 9091 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:10.037451 9106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.037722 9091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:10.037856 9091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "d34aa3ff7353410abc65aaae470dcca7"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "3eb07f88528a1b00662b6f32e6abcb5d"
server_key_iv: "c85c9f9f64c337c054084ececb2d63c9"
server_key_version: "encryptionkey@0"
I20260926 13:15:10.038300 9091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:10.066251 9091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:10.068372 9091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:10.068572 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:10.072587 9091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34135
I20260926 13:15:10.072638 9158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34135 every 8 connection(s)
I20260926 13:15:10.073231 9091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:15:10.074589 9159 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:15:10.074988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9091
I20260926 13:15:10.075290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41141
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34135
--webserver_port=32849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:15:10.075573 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Bootstrap starting.
I20260926 13:15:10.083920 9159 log.cc:824] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:10.085829 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:15:10.086133 9159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Bootstrap complete.
I20260926 13:15:10.088411 9159 raft_consensus.cc:348] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:10.088666 9159 raft_consensus.cc:729] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d34aa3ff7353410abc65aaae470dcca7, State: Initialized, Role: FOLLOWER
I20260926 13:15:10.088845 9159 consensus_queue.cc:260] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:10.088917 9159 raft_consensus.cc:388] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:15:10.088970 9159 raft_consensus.cc:482] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:15:10.089017 9159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:15:10.089982 9159 raft_consensus.cc:504] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:10.090065 9159 leader_election.cc:302] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d34aa3ff7353410abc65aaae470dcca7; no voters:
I20260926 13:15:10.090312 9159 leader_election.cc:288] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:15:10.090353 9163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:15:10.090474 9163 raft_consensus.cc:686] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [term 2 LEADER]: Becoming Leader. State: Replica: d34aa3ff7353410abc65aaae470dcca7, State: Running, Role: LEADER
I20260926 13:15:10.090559 9163 consensus_queue.cc:237] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } }
I20260926 13:15:10.090582 9159 sys_catalog.cc:564] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:15:10.090826 9163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d34aa3ff7353410abc65aaae470dcca7. Latest consensus state: current_term: 2 leader_uuid: "d34aa3ff7353410abc65aaae470dcca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } } }
I20260926 13:15:10.090900 9163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:10.090831 9164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d34aa3ff7353410abc65aaae470dcca7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d34aa3ff7353410abc65aaae470dcca7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34135 } } }
I20260926 13:15:10.091063 9164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:10.091159 9167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:15:10.091676 9167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:15:10.091838 9167 catalog_manager.cc:1259] Loaded cluster ID: ddfad7b53a9049d1b8bc0573c192696d
I20260926 13:15:10.092124 9167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:15:10.093058 9167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:15:10.093250 9167 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d34aa3ff7353410abc65aaae470dcca7: Loaded TSK: 0
I20260926 13:15:10.093465 9167 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:15:10.163722 9161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:10.163991 9161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:10.164026 9161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:10.166031 9161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:10.166085 9161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:10.166141 9161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:15:10.168942 9161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32849
--tserver_master_addrs=127.0.98.62:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:10.169189 9161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:10.169433 9161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:10.175552 9188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.175529 9185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.175542 9186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:10.175803 9161 server_base.cc:1034] running on GCE node
I20260926 13:15:10.176085 9161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:10.176373 9161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:10.177537 9161 hybrid_clock.cc:648] HybridClock initialized: now 1790428510177524 us; error 36 us; skew 500 ppm
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428510, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:10.180423 9161 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 13:15:10.180963 9161 webserver.cc:460] Webserver started at http://127.0.98.1:32849/ using document root <none> and password file <none>
I20260926 13:15:10.181223 9161 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:10.181311 9161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:10.182850 9161 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:10.183503 9195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.183704 9161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:15:10.183817 9161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "6a7c87fc6a1242c98159c552199e9ee5"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "e8974e91ae787479b0c98f391da3cafc"
server_key_iv: "c7c37e3cad71caaf5ef972786d75351c"
server_key_version: "encryptionkey@0"
I20260926 13:15:10.184185 9161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:10.198521 9161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:10.201611 9161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:10.201777 9161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:10.202039 9161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:10.202436 9161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:10.202488 9161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.202520 9161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:10.202538 9161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.210003 9161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41141
I20260926 13:15:10.210047 9308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41141 every 8 connection(s)
I20260926 13:15:10.210485 9161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:15:10.210732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9161
I20260926 13:15:10.211045 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46703
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34135
--webserver_port=46553
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.224807 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.213219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:45709)
0926 13:15:10.213750 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.213800 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.214656 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.215167 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.215172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.216160 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.216508 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.216517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.218210 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.218212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.219812 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.219816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.220633 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.220635 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.220817 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.221533 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.221545 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.224583 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":94,"threads_started":1}
W20260926 13:15:10.225060 9310 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34135 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34135: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260926 13:15:10.225189 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.213289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45709 (local address 127.0.98.62:34135)
0926 13:15:10.213528 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.213540 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.213946 (+ 406us) server_negotiation.cc:407] Connection header received
0926 13:15:10.214872 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.214879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.214944 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.215082 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.216646 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.218060 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.219921 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.220522 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.220634 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.224728 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45709: BlockingRecv error: recv got EOF from 127.0.98.1:45709 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.230996 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.225280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:37371)
0926 13:15:10.225502 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.225514 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.225559 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.225719 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.225720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.225877 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.225933 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.225935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.226986 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.226987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.227818 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.227820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.228326 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.228327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.228393 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.228894 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.228900 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.230912 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":101,"threads_started":1}
W20260926 13:15:10.231619 9313 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.225494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37371 (local address 127.0.98.62:34135)
0926 13:15:10.225529 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.225531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.225543 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:15:10.225624 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.225625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.225636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.225696 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.226026 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.226904 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.227905 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.228239 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.228309 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.231033 (+ 2724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37371: BlockingRecv error: recv got EOF from 127.0.98.1:37371 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.236526 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.231393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:37943)
0926 13:15:10.231439 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.231462 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.231503 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.231799 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.231800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.231955 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.231989 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.231990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.232997 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.232997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.233476 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.233477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.233973 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.233973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.234011 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.234584 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.234589 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.236473 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:15:10.236711 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.231450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37943 (local address 127.0.98.62:34135)
0926 13:15:10.231635 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.231637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.231646 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:15:10.231667 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.231668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.231679 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.231754 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.232070 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.232917 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.233551 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.233892 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.233968 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.236583 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37943: BlockingRecv error: recv got EOF from 127.0.98.1:37943 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":115,"threads_started":1}
W20260926 13:15:10.236719 9310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:15:10.304436 9309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:10.304697 9309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:10.304754 9309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:10.305972 9309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:10.306015 9309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:10.306041 9309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:15:10.307487 9309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46553
--tserver_master_addrs=127.0.98.62:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:10.307741 9309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:10.307966 9309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:10.313045 9323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.313045 9320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:10.313263 9309 server_base.cc:1034] running on GCE node
W20260926 13:15:10.313318 9321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:10.313544 9309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:10.313742 9309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:10.314903 9309 hybrid_clock.cc:648] HybridClock initialized: now 1790428510314891 us; error 32 us; skew 500 ppm
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428510, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:10.317467 9309 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 13:15:10.317905 9309 webserver.cc:460] Webserver started at http://127.0.98.2:46553/ using document root <none> and password file <none>
I20260926 13:15:10.318147 9309 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:10.318261 9309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:10.319680 9309 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:10.320264 9330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.320542 9309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:10.320683 9309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "f034c99b632445ed864ad528403c7def"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "52a62213339c1cd565c20ab2876e8b4a"
server_key_iv: "cfe42f60397fa80dc3641b7738a6449a"
server_key_version: "encryptionkey@0"
I20260926 13:15:10.321089 9309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:10.329869 9309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:10.332218 9309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:10.332434 9309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:10.332707 9309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:10.333067 9309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:10.333128 9309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.333158 9309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:10.333175 9309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.340101 9309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46703
I20260926 13:15:10.340194 9443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46703 every 8 connection(s)
I20260926 13:15:10.340564 9309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.347297 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9309
I20260926 13:15:10.347594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45161
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34135
--webserver_port=46643
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:15:10.348881 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.341517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49925 (local address 127.0.98.62:34135)
0926 13:15:10.341572 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.341575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.341789 (+ 214us) server_negotiation.cc:407] Connection header received
0926 13:15:10.342398 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.342399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.342411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.342476 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.343464 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.343944 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.344991 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.345174 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.345227 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.348803 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49925: BlockingRecv error: recv got EOF from 127.0.98.2:49925 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:15:10.348925 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.341471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:49925)
0926 13:15:10.341643 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.341689 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.342301 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.342534 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.342538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.343167 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.343379 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.343385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.344031 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.344032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.344901 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.344903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.345250 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.345251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.345357 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.345882 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.345890 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.348700 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
W20260926 13:15:10.349157 9444 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34135 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34135: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.354709 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.349431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:36143)
0926 13:15:10.349491 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.349504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.349556 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.349727 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.349728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.349882 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.349937 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.349939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.350772 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.350773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.351345 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.351347 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.351797 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.351798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.351882 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.352431 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.352437 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.354654 (+ 2217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 13:15:10.354866 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.349497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36143 (local address 127.0.98.62:34135)
0926 13:15:10.349550 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.349552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.349564 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:15:10.349637 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.349638 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.349653 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.349722 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.350032 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.350658 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.351463 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.351695 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.351796 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.354783 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36143: BlockingRecv error: recv got EOF from 127.0.98.2:36143 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.360118 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.355230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:35895)
0926 13:15:10.355282 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.355294 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.355339 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.355480 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.355480 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.355622 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.355652 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.355653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.356383 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.356384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.356927 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.356928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.357459 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.357460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.357503 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.358001 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.358009 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.360042 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:15:10.360265 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.355299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35895 (local address 127.0.98.62:34135)
0926 13:15:10.355345 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.355348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.355373 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:15:10.355398 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.355399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.355413 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.355475 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.355740 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.356273 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.357033 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.357256 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.357345 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.360144 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35895: BlockingRecv error: recv got EOF from 127.0.98.2:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 13:15:10.360426 9444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:15:10.439298 9447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:10.439524 9447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:10.439550 9447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:10.440799 9447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:10.440841 9447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:10.440867 9447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:15:10.442396 9447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46643
--tserver_master_addrs=127.0.98.62:34135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:10.442595 9447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:10.442776 9447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:10.447752 9453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.447757 9452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:10.447732 9455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:10.447903 9447 server_base.cc:1034] running on GCE node
I20260926 13:15:10.448225 9447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:10.448438 9447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:10.449616 9447 hybrid_clock.cc:648] HybridClock initialized: now 1790428510449595 us; error 34 us; skew 500 ppm
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428510, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:10.452205 9447 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 13:15:10.452745 9447 webserver.cc:460] Webserver started at http://127.0.98.3:46643/ using document root <none> and password file <none>
I20260926 13:15:10.453011 9447 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:10.453114 9447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:10.454640 9447 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:15:10.455339 9462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.455570 9447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:10.455689 9447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "a77c4de56a114d3d9b0645ca80822b32"
format_stamp: "Formatted at 2026-09-26 13:15:09 on dist-test-slave-f864"
server_key: "17552a594c15d50994dd34e4eee925ef"
server_key_iv: "5afe59713782d17877d0a9ed8a4e588e"
server_key_version: "encryptionkey@0"
I20260926 13:15:10.456074 9447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:10.474557 9447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:10.476648 9447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:10.476819 9447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:10.477077 9447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:10.477389 9447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:10.477445 9447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.477488 9447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:10.477517 9447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:10.483559 9447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45161
I20260926 13:15:10.483608 9575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45161 every 8 connection(s)
I20260926 13:15:10.483954 9447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 13:15:10.491964 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.484709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57611 (local address 127.0.98.62:34135)
0926 13:15:10.484757 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.484759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.485039 (+ 280us) server_negotiation.cc:407] Connection header received
0926 13:15:10.485588 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.485589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.485603 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.485659 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.486697 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.487191 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.488249 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.488450 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.488535 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.491914 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57611: BlockingRecv error: recv got EOF from 127.0.98.3:57611 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:15:10.492043 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.484720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:57611)
0926 13:15:10.484914 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.484949 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.485465 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.485741 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.485746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.486394 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.486591 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.486596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.487295 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.487297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.488140 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.488142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.488544 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.488545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.488668 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.489156 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.489165 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.491833 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":98,"threads_started":1}
W20260926 13:15:10.492286 9576 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34135 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:34135: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260926 13:15:10.493132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9447
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.497687 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.492515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:49647)
0926 13:15:10.492572 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.492585 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.492636 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.492880 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.492882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.493060 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.493119 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.493120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.494056 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.494057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.494681 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.494683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.495104 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.495105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.495184 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.495674 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.495679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.497636 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:15:10.497824 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.492725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49647 (local address 127.0.98.62:34135)
0926 13:15:10.492767 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.492769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.492775 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:15:10.492794 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.492795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.492803 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.492874 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.493204 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.493943 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.494785 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.495012 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.495105 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.497754 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49647: BlockingRecv error: recv got EOF from 127.0.98.3:49647 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 13:15:10.501142 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.493558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.0.98.62:34135)
0926 13:15:10.493616 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.493618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.493777 (+ 159us) server_negotiation.cc:407] Connection header received
0926 13:15:10.493860 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.493860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.493872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.493929 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.494574 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.495031 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.495789 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.495982 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.496066 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.497471 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:10.497476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:10.497477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:10.497490 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:10.499560 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:10.499952 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:10.499955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:10.499956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:10.500098 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:10.500340 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:10.500341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:10.500341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:10.500675 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:10.500786 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:10.500932 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 13:15:10.501022 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:10 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:10.502591 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:10.498009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:58341)
0926 13:15:10.498095 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:10.498105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:10.498148 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:10.498374 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:10.498374 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:10.498517 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:10.498543 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.498544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.499149 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.499151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.499686 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:10.499688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.500147 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.500148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:10.500206 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.500608 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:10.500612 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:10.502547 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 13:15:10.502664 9315 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:10.498057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58341 (local address 127.0.98.62:34135)
0926 13:15:10.498097 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:10.498099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:10.498138 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:15:10.498284 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:10.498286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:10.498297 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:10.498372 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:10.498621 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.499071 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.499779 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:10.500046 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:10.500116 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:10.502627 (+ 2511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58341: BlockingRecv error: recv got EOF from 127.0.98.3:58341 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:15:10.502810 9576 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 13:15:11 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:11 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:11.241814 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:11.237097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:55445)
0926 13:15:11.237256 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:11.237266 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:11.237317 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:11.237458 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:11.237459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:11.237611 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:11.237682 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:11.237683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.238439 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.238440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:11.238962 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:11.238964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.239280 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.239281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:11.239327 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:11.239837 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:11.239842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:11.241706 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
W20260926 13:15:11.241871 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:11.237170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55445 (local address 127.0.98.62:34135)
0926 13:15:11.237293 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:11.237295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:11.237305 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:15:11.237366 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:11.237367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:11.237379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:11.237438 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:11.237742 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.238368 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.239028 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.239216 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.239269 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:11.241787 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55445: BlockingRecv error: recv got EOF from 127.0.98.1:55445 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
Sep 26 13:15:11 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:11 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:11.365609 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:11.360900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:57997)
0926 13:15:11.361040 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:11.361051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:11.361118 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:11.361266 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:11.361267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:11.361423 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:11.361472 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:11.361474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.362132 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.362133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:11.362721 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:11.362722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.363245 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.363246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:11.363312 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:11.363772 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:11.363777 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:11.365504 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
W20260926 13:15:11.365639 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:11.360968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57997 (local address 127.0.98.62:34135)
0926 13:15:11.361074 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:11.361076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:11.361113 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:15:11.361177 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:11.361178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:11.361190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:11.361247 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:11.361553 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.362056 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.362827 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.363170 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.363239 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:11.365577 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57997: BlockingRecv error: recv got EOF from 127.0.98.2:57997 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 13:15:11 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:11 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:11.507926 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:11.503295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:36131)
0926 13:15:11.503501 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:11.503515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:11.503602 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:11.503774 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:11.503775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:11.503998 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:11.504056 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:11.504059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.504766 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.504767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:11.505302 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:11.505303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.505626 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.505627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:11.505664 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:11.506209 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:11.506213 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:11.507854 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
W20260926 13:15:11.507985 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:11.503418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36131 (local address 127.0.98.62:34135)
0926 13:15:11.503506 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:11.503508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:11.503622 (+ 114us) server_negotiation.cc:407] Connection header received
0926 13:15:11.503661 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:11.503661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:11.503676 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:11.503740 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:11.504159 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.504695 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.505379 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:11.505561 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:11.505614 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:11.507939 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36131: BlockingRecv error: recv got EOF from 127.0.98.3:36131 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 13:15:12 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:12 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:12.247081 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:12.242366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:37629)
0926 13:15:12.242511 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:12.242522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:12.242588 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:12.242736 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:12.242744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:12.242928 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:12.242981 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:12.242983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.243631 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.243632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:12.244173 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:12.244174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.244507 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.244508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:12.244554 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:12.245142 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:12.245147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:12.247019 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
W20260926 13:15:12.247241 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:12.242437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37629 (local address 127.0.98.62:34135)
0926 13:15:12.242569 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:12.242571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:12.242579 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:12.242651 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:12.242652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:12.242662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:12.242717 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:12.243047 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.243542 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.244251 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.244437 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.244488 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:12.247150 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37629: BlockingRecv error: recv got EOF from 127.0.98.1:37629 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
Sep 26 13:15:12 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:12 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:12.371197 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:12.366129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:44069)
0926 13:15:12.366297 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:12.366311 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:12.366389 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:12.366570 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:12.366571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:12.366798 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:12.366849 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:12.366852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.367586 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.367587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:12.368142 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:12.368144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.368554 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.368555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:12.368616 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:12.369197 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:12.369218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:12.371141 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
W20260926 13:15:12.371397 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:12.366337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44069 (local address 127.0.98.62:34135)
0926 13:15:12.366386 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:12.366388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:12.366399 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:15:12.366462 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:12.366463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:12.366478 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:12.366562 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:12.366943 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.367488 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.368266 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.368461 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.368554 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:12.371299 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44069: BlockingRecv error: recv got EOF from 127.0.98.2:44069 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 13:15:12 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:12 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:12.513295 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:12.508368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:44705)
0926 13:15:12.508529 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:12.508542 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:12.508636 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:12.508773 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:12.508781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:12.508963 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:12.509007 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:12.509009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.509696 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.509697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:12.510314 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:12.510317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.510701 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.510702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:12.510769 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:12.511292 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:12.511297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:12.513237 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
W20260926 13:15:12.513417 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:12.508563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44705 (local address 127.0.98.62:34135)
0926 13:15:12.508628 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:12.508630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:12.508639 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:15:12.508697 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:12.508698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:12.508712 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:12.508759 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:12.509090 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.509605 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.510398 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:12.510596 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:12.510699 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:12.513340 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44705: BlockingRecv error: recv got EOF from 127.0.98.3:44705 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 13:15:13 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:13 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:13.252360 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:13.247552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:48699)
0926 13:15:13.247722 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:13.247733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:13.247784 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:13.247948 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:13.247949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:13.248108 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:13.248158 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:13.248159 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.248791 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.248792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:13.249336 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:13.249338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.249686 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.249687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:13.249732 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:13.250354 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:13.250359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:13.252265 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
W20260926 13:15:13.252408 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:13.247643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48699 (local address 127.0.98.62:34135)
0926 13:15:13.247776 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:13.247777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:13.247787 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:15:13.247842 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:13.247843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:13.247852 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:13.247925 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:13.248223 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.248712 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.249414 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.249602 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.249653 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:13.252341 (+ 2688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48699: BlockingRecv error: recv got EOF from 127.0.98.1:48699 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
Sep 26 13:15:13 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:13 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:13.376399 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:13.371696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:34957)
0926 13:15:13.371876 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:13.371888 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:13.371981 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:13.372133 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:13.372134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:13.372312 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:13.372359 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:13.372361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.373047 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.373048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:13.373575 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:13.373577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.373947 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.373948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:13.374004 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:13.374528 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:13.374532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:13.376295 (+ 1763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":96,"threads_started":1}
W20260926 13:15:13.376457 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:13.371789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34957 (local address 127.0.98.62:34135)
0926 13:15:13.371860 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:13.371863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:13.371967 (+ 104us) server_negotiation.cc:407] Connection header received
0926 13:15:13.372040 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:13.372041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:13.372053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:13.372129 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:13.372453 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.372963 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.373663 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.373870 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.373950 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:13.376387 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34957: BlockingRecv error: recv got EOF from 127.0.98.2:34957 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 13:15:13 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:13 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:13.518086 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:13.513796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:35417)
0926 13:15:13.513945 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:13.513956 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:13.514028 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:13.514202 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:13.514203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:13.514386 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:13.514433 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:13.514435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.515061 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.515062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:13.515609 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:13.515611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.515916 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.515917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:13.515956 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:13.516380 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:13.516385 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:13.518025 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260926 13:15:13.518153 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:13.513863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35417 (local address 127.0.98.62:34135)
0926 13:15:13.513908 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:13.513910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:13.514013 (+ 103us) server_negotiation.cc:407] Connection header received
0926 13:15:13.514083 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:13.514084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:13.514095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:13.514137 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:13.514509 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.514986 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.515683 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:13.515854 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:13.515903 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:13.518106 (+ 2203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35417: BlockingRecv error: recv got EOF from 127.0.98.3:35417 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 26 13:15:14 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:14 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:14.262537 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:14.253088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:47267)
0926 13:15:14.253260 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:14.253271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:14.253330 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:14.253491 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:14.253492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:14.253649 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:14.253697 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:14.253699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.254441 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.254442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:14.255178 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:14.255181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.255509 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.255510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:14.255549 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:14.260635 (+ 5086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:14.260641 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:14.262474 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":87,"threads_started":1}
W20260926 13:15:14.262744 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:14.253211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47267 (local address 127.0.98.62:34135)
0926 13:15:14.253349 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:14.253351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:14.253360 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:15:14.253383 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:14.253384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:14.253394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:14.253452 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:14.253778 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.254321 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.255242 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.255440 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.255492 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:14.262608 (+ 7116us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47267: BlockingRecv error: recv got EOF from 127.0.98.1:47267 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
Sep 26 13:15:14 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:14 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:14.381549 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:14.376938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:40605)
0926 13:15:14.377084 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:14.377095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:14.377146 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:14.377283 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:14.377284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:14.377441 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:14.377491 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:14.377493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.378142 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.378143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:14.378714 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:14.378716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.379102 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.379103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:14.379144 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:14.379603 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:14.379608 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:14.381489 (+ 1881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260926 13:15:14.381718 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:14.377014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40605 (local address 127.0.98.62:34135)
0926 13:15:14.377134 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:14.377137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:14.377147 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:15:14.377196 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:14.377196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:14.377207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:14.377271 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:14.377557 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.378066 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.378816 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.379020 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.379096 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:14.381617 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40605: BlockingRecv error: recv got EOF from 127.0.98.2:40605 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 13:15:14 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:14 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:14.522967 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:14.518580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:50493)
0926 13:15:14.518717 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:14.518729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:14.518790 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:14.518910 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:14.518910 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:14.519061 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:14.519107 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:14.519109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.519708 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.519709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:14.520218 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:14.520219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.520546 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.520547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:14.520581 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:14.521029 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:14.521034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:14.522908 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260926 13:15:14.523066 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:14.518699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50493 (local address 127.0.98.62:34135)
0926 13:15:14.518781 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:14.518783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:14.518791 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:14.518835 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:14.518836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:14.518847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:14.518894 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:14.519164 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.519647 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.520289 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:14.520470 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:14.520525 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:14.523002 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50493: BlockingRecv error: recv got EOF from 127.0.98.3:50493 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 13:15:15 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:15 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:15.267858 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:15.263073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:54801)
0926 13:15:15.263194 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:15.263209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:15.263271 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:15.263414 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:15.263415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:15.263576 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:15.263626 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:15.263628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.264361 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.264369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:15.264896 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:15.264897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.265303 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.265304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:15.265346 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:15.265870 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:15.265875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:15.267797 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
W20260926 13:15:15.267969 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:15.263155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54801 (local address 127.0.98.62:34135)
0926 13:15:15.263295 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:15.263297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:15.263304 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:15:15.263326 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:15.263327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:15.263337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:15.263397 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:15.263737 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.264280 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.264992 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.265228 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.265288 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:15.267891 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54801: BlockingRecv error: recv got EOF from 127.0.98.1:54801 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
Sep 26 13:15:15 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:15 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:15.386849 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:15.382088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:48957)
0926 13:15:15.382254 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:15.382271 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:15.382354 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:15.382521 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:15.382522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:15.382713 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:15.382762 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:15.382764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.383451 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.383453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:15.383997 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:15.383998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.384362 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.384364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:15.384427 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:15.384882 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:15.384887 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:15.386737 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260926 13:15:15.386893 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:15.382275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48957 (local address 127.0.98.62:34135)
0926 13:15:15.382351 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:15.382353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:15.382364 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:15:15.382419 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:15.382420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:15.382438 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:15.382499 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:15.382857 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.383364 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.384089 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.384288 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.384364 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:15.386826 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48957: BlockingRecv error: recv got EOF from 127.0.98.2:48957 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 13:15:15 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:15 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:15.528069 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:15.523421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:45811)
0926 13:15:15.523563 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:15.523581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:15.523633 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:15.523776 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:15.523777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:15.523930 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:15.523991 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:15.523993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.524644 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.524645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:15.525234 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:15.525236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.525577 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.525578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:15.525633 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:15.526110 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:15.526116 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:15.528002 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
W20260926 13:15:15.528187 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:15.523480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45811 (local address 127.0.98.62:34135)
0926 13:15:15.523544 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:15.523546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:15.523653 (+ 107us) server_negotiation.cc:407] Connection header received
0926 13:15:15.523679 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:15.523679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:15.523691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:15.523772 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:15.524051 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.524547 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.525319 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:15.525516 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:15.525569 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:15.528110 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45811: BlockingRecv error: recv got EOF from 127.0.98.3:45811 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 13:15:16 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:16 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:16.273866 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:16.268549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:46097)
0926 13:15:16.268750 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:16.268764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:16.268830 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:16.269031 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:16.269032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:16.269280 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:16.269339 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:16.269342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.270010 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.270011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:16.270563 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:16.270564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.271017 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.271018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:16.271063 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:16.271511 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:16.271515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:16.273772 (+ 2257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
W20260926 13:15:16.274047 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:16.268598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46097 (local address 127.0.98.62:34135)
0926 13:15:16.268732 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:16.268735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:16.268841 (+ 106us) server_negotiation.cc:407] Connection header received
0926 13:15:16.268899 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:16.268900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:16.268929 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:16.269002 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:16.269424 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.269935 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.270634 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.270940 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.270997 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:16.273937 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46097: BlockingRecv error: recv got EOF from 127.0.98.1:46097 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
Sep 26 13:15:16 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:16 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:16.393709 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:16.387439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:36773)
0926 13:15:16.387584 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:16.387598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:16.387670 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:16.387865 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:16.387865 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:16.388224 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:16.388322 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:16.388326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.389253 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.389254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:16.389853 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:16.389855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.390361 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.390362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:16.390426 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:16.391040 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:16.391045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:16.393622 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
W20260926 13:15:16.393846 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:16.387529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36773 (local address 127.0.98.62:34135)
0926 13:15:16.387630 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:16.387633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:16.387648 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:15:16.387773 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:16.387773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:16.387788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:16.387845 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:16.388419 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.389161 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.389946 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.390259 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.390353 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:16.393764 (+ 3411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36773: BlockingRecv error: recv got EOF from 127.0.98.2:36773 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 13:15:16 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:16 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:16.533380 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:16.528592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:38515)
0926 13:15:16.528787 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:16.528798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:16.528887 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:16.529059 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:16.529060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:16.529236 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:16.529283 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:16.529285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.529945 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.529946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:16.530529 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:16.530531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.530896 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.530897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:16.530957 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:16.531560 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:16.531566 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:16.533285 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
W20260926 13:15:16.533419 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:16.528801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38515 (local address 127.0.98.62:34135)
0926 13:15:16.528865 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:16.528868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:16.528879 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:15:16.528956 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:16.528957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:16.528973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:16.529053 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:16.529371 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.529867 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.530623 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:16.530814 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:16.530897 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:16.533357 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38515: BlockingRecv error: recv got EOF from 127.0.98.3:38515 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 13:15:17 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:17 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:17.279732 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:17.274565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:52613)
0926 13:15:17.274685 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:17.274698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:17.274786 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:17.274919 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:17.274920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:17.275078 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:17.275127 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:17.275128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.275770 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.275770 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:17.276330 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:17.276333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.276848 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.276849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:17.276922 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:17.277531 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:17.277537 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:17.279631 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
W20260926 13:15:17.279808 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:17.274628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52613 (local address 127.0.98.62:34135)
0926 13:15:17.274749 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:17.274751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:17.274765 (+ 14us) server_negotiation.cc:407] Connection header received
0926 13:15:17.274834 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:17.274834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:17.274844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:17.274911 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:17.275195 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.275703 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.276421 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.276754 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.276848 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:17.279733 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52613: BlockingRecv error: recv got EOF from 127.0.98.1:52613 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
Sep 26 13:15:17 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:17 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:17.398975 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:17.394278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:59709)
0926 13:15:17.394439 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:17.394451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:17.394537 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:17.394701 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:17.394702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:17.394888 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:17.394935 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:17.394937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.395605 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.395606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:17.396136 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:17.396138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.396513 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.396514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:17.396578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:17.397092 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:17.397097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:17.398880 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
W20260926 13:15:17.399027 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:17.394350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59709 (local address 127.0.98.62:34135)
0926 13:15:17.394424 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:17.394426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:17.394522 (+ 96us) server_negotiation.cc:407] Connection header received
0926 13:15:17.394610 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:17.394611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:17.394625 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:17.394694 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:17.395016 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.395522 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.396236 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.396432 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.396514 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:17.398964 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59709: BlockingRecv error: recv got EOF from 127.0.98.2:59709 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 13:15:17 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:17 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:17.538301 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:17.533945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:51587)
0926 13:15:17.534087 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:17.534096 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:17.534156 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:17.534360 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:17.534361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:17.534516 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:17.534564 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:17.534565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.535184 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.535185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:17.535700 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:17.535701 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.536005 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.536006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:17.536043 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:17.536478 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:17.536482 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:17.538241 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
W20260926 13:15:17.538415 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:17.534017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51587 (local address 127.0.98.62:34135)
0926 13:15:17.534081 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:17.534088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:17.534153 (+ 65us) server_negotiation.cc:407] Connection header received
0926 13:15:17.534275 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:17.534275 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:17.534287 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:17.534340 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:17.534629 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.535121 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.535766 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:17.535944 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:17.535996 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:17.538322 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51587: BlockingRecv error: recv got EOF from 127.0.98.3:51587 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 13:15:18 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:18 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:18.286307 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:18.280388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:33079)
0926 13:15:18.280579 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:18.280592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:18.280657 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:18.280891 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:18.280892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:18.281144 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:18.281207 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:18.281210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.282209 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.282210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:18.283097 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:18.283101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.283593 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.283594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:18.283664 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:18.284156 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:18.284161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:18.286242 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":85,"threads_started":1}
W20260926 13:15:18.286412 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:18.280510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33079 (local address 127.0.98.62:34135)
0926 13:15:18.280648 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:18.280651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:18.280663 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:15:18.280765 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:18.280766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:18.280781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:18.280862 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:18.281317 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.282095 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.283192 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.283492 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.283580 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:18.286324 (+ 2744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33079: BlockingRecv error: recv got EOF from 127.0.98.1:33079 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":54,"threads_started":1}
Sep 26 13:15:18 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:18 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:18.404629 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:18.399521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:46973)
0926 13:15:18.399660 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:18.399671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:18.399734 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:18.399866 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:18.399867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:18.400022 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:18.400079 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:18.400081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.400904 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.400905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:18.401455 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:18.401456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.401853 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.401854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:18.401907 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:18.402620 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:18.402626 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:18.404534 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260926 13:15:18.404744 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:18.399569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46973 (local address 127.0.98.62:34135)
0926 13:15:18.399625 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:18.399627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:18.399735 (+ 108us) server_negotiation.cc:407] Connection header received
0926 13:15:18.399792 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:18.399793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:18.399804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:18.399850 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:18.400140 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.400829 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.401548 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.401759 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.401824 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:18.404660 (+ 2836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46973: BlockingRecv error: recv got EOF from 127.0.98.2:46973 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 13:15:18 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:18 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:18.543300 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:18.538799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:34581)
0926 13:15:18.539000 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:18.539011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:18.539078 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:18.539217 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:18.539218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:18.539393 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:18.539447 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:18.539448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.540063 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.540063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:18.540586 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:18.540587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.540991 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.540993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:18.541073 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:18.541570 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:18.541575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:18.543242 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
W20260926 13:15:18.543442 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:18.538879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34581 (local address 127.0.98.62:34135)
0926 13:15:18.538939 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:18.538941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:18.539089 (+ 148us) server_negotiation.cc:407] Connection header received
0926 13:15:18.539134 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:18.539135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:18.539149 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:18.539194 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:18.539512 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.539997 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.540675 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:18.540895 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:18.540955 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:18.543350 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34581: BlockingRecv error: recv got EOF from 127.0.98.3:34581 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 13:15:19 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:19 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:19.300590 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:19.286835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:51453)
0926 13:15:19.287003 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:19.287015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:19.287100 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:19.287446 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:19.287447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:19.287610 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:19.287661 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:19.287663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.288445 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.288446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:19.289248 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:19.289250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.289594 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.289594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:19.289639 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:19.298149 (+ 8510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:19.298157 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:19.300528 (+ 2371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
W20260926 13:15:19.300803 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:19.286879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51453 (local address 127.0.98.62:34135)
0926 13:15:19.287254 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:19.287256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:19.287292 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:15:19.287327 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:19.287327 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:19.287340 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:19.287430 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:19.287755 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.288346 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.289313 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.289531 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.289589 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:19.300728 (+ 11139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51453: BlockingRecv error: recv got EOF from 127.0.98.1:51453 (error 108)
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":77,"threads_started":1}
Sep 26 13:15:19 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:19 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:19.410076 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:19.405164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:47111)
0926 13:15:19.405357 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:19.405372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:19.405476 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:19.405659 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:19.405660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:19.405819 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:19.405869 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:19.405871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.406532 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.406533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:19.407100 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:19.407102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.407416 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.407416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:19.407454 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:19.408061 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:19.408066 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:19.410018 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260926 13:15:19.410248 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:19.405292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47111 (local address 127.0.98.62:34135)
0926 13:15:19.405358 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:19.405360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:19.405460 (+ 100us) server_negotiation.cc:407] Connection header received
0926 13:15:19.405546 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:19.405546 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:19.405557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:19.405607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:19.405940 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.406452 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.407174 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.407356 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.407406 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:19.410147 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47111: BlockingRecv error: recv got EOF from 127.0.98.2:47111 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 13:15:19 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:19 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:19.548621 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:19.543811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:48817)
0926 13:15:19.543948 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:19.543960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:19.544039 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:19.544203 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:19.544204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:19.544439 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:19.544505 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:19.544508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.545309 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.545310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:19.545848 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:19.545849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.546225 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.546226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:19.546291 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:19.546718 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:19.546723 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:19.548550 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260926 13:15:19.548792 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:19.543869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48817 (local address 127.0.98.62:34135)
0926 13:15:19.543912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:19.543915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:19.544044 (+ 129us) server_negotiation.cc:407] Connection header received
0926 13:15:19.544106 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:19.544106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:19.544118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:19.544190 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:19.544589 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.545231 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.545917 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:19.546109 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:19.546181 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:19.548708 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48817: BlockingRecv error: recv got EOF from 127.0.98.3:48817 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 13:15:20 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:20 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:20.305817 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:20.301136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:55773)
0926 13:15:20.301284 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:20.301297 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:20.301370 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:20.301586 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:20.301587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:20.301793 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:20.301853 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:20.301855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.302658 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.302658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:20.303183 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:20.303184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.303505 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.303506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:20.303544 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:20.303970 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:20.303974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:20.305721 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
W20260926 13:15:20.305874 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:20.301199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55773 (local address 127.0.98.62:34135)
0926 13:15:20.301338 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:20.301340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:20.301353 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:15:20.301444 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:20.301445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:20.301461 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:20.301543 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:20.301957 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.302590 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.303256 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.303443 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.303498 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:20.305802 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55773: BlockingRecv error: recv got EOF from 127.0.98.1:55773 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
Sep 26 13:15:20 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:20 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:20.415094 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:20.410543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:37341)
0926 13:15:20.410671 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:20.410682 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:20.410749 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:20.410895 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:20.410895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:20.411056 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:20.411101 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:20.411102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.411762 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.411764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:20.412316 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:20.412318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.412686 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.412687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:20.412726 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:20.413176 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:20.413181 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:20.415021 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
W20260926 13:15:20.415238 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:20.410580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37341 (local address 127.0.98.62:34135)
0926 13:15:20.410626 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:20.410628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:20.410784 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:15:20.410811 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:20.410812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:20.410824 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:20.410871 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:20.411168 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.411677 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.412415 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.412617 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.412674 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:20.415150 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37341: BlockingRecv error: recv got EOF from 127.0.98.2:37341 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 26 13:15:20 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:20 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:20.553866 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:20.549122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:41433)
0926 13:15:20.549274 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:20.549285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:20.549371 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:20.549536 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:20.549537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:20.549710 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:20.549755 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:20.549757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.550504 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.550505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:20.551037 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:20.551039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.551409 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.551410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:20.551469 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:20.551908 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:20.551912 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:20.553807 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":58,"threads_started":1}
W20260926 13:15:20.553979 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:20.549289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41433 (local address 127.0.98.62:34135)
0926 13:15:20.549357 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:20.549359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:20.549367 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:20.549431 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:20.549432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:20.549447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:20.549528 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:20.549839 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.550399 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.551131 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:20.551311 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:20.551390 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:20.553892 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41433: BlockingRecv error: recv got EOF from 127.0.98.3:41433 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 13:15:21 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:21 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:21.310700 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:21.306308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:45997)
0926 13:15:21.306454 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:21.306465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:21.306525 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:21.306640 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:21.306640 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:21.306811 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:21.306859 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:21.306860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.307474 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.307475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:21.308007 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:21.308008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.308329 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.308329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:21.308377 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:21.308763 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:21.308768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:21.310608 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
W20260926 13:15:21.310750 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:21.306379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45997 (local address 127.0.98.62:34135)
0926 13:15:21.306475 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:21.306477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:21.306517 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:15:21.306578 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:21.306579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:21.306589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:21.306634 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:21.306920 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.307407 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.308082 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.308268 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.308319 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:21.310682 (+ 2363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45997: BlockingRecv error: recv got EOF from 127.0.98.1:45997 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":46,"threads_started":1}
Sep 26 13:15:21 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:21 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:21.419888 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:21.415550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:52581)
0926 13:15:21.415683 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:21.415693 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:21.415747 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:21.415867 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:21.415868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:21.416027 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:21.416076 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:21.416077 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.416690 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.416691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:21.417221 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:21.417222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.417523 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.417524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:21.417562 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:21.418001 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:21.418005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:21.419793 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
W20260926 13:15:21.419948 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:21.415633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52581 (local address 127.0.98.62:34135)
0926 13:15:21.415690 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:21.415691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:21.415737 (+ 46us) server_negotiation.cc:407] Connection header received
0926 13:15:21.415791 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:21.415791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:21.415801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:21.415850 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:21.416135 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.416631 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.417290 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.417465 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.417513 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:21.419873 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52581: BlockingRecv error: recv got EOF from 127.0.98.2:52581 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 13:15:21 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:21 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:21.559263 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:21.554450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:43091)
0926 13:15:21.554580 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:21.554593 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:21.554670 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:21.554864 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:21.554865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:21.555051 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:21.555099 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:21.555100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.555774 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.555775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:21.556311 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:21.556313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.556698 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.556699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:21.556765 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:21.557355 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:21.557360 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:21.559203 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260926 13:15:21.559453 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:21.554553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43091 (local address 127.0.98.62:34135)
0926 13:15:21.554596 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:21.554598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:21.554649 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:15:21.554757 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:21.554758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:21.554772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:21.554840 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:21.555183 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.555688 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.556419 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:21.556612 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:21.556696 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:21.559373 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43091: BlockingRecv error: recv got EOF from 127.0.98.3:43091 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 13:15:22 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:22 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:22.315824 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:22.311196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:56003)
0926 13:15:22.311335 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:22.311351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:22.311403 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:22.311552 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:22.311553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:22.311719 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:22.311766 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:22.311768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.312398 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.312398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:22.312910 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:22.312912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.313218 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.313219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:22.313258 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:22.313745 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:22.313750 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:22.315736 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
W20260926 13:15:22.316042 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:22.311290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56003 (local address 127.0.98.62:34135)
0926 13:15:22.311413 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:22.311415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:22.311425 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:15:22.311456 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:22.311457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:22.311467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:22.311539 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:22.311827 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.312332 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.312972 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.313159 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.313208 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:22.315873 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56003: BlockingRecv error: recv got EOF from 127.0.98.1:56003 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
Sep 26 13:15:22 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:22 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:22.425089 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:22.420426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:35259)
0926 13:15:22.420600 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:22.420612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:22.420703 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:22.420853 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:22.420854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:22.421034 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:22.421079 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:22.421080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.421735 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.421736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:22.422295 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:22.422296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.422659 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.422660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:22.422722 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:22.423195 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:22.423199 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:22.425032 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
W20260926 13:15:22.425207 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:22.420524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35259 (local address 127.0.98.62:34135)
0926 13:15:22.420602 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:22.420604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:22.420685 (+ 81us) server_negotiation.cc:407] Connection header received
0926 13:15:22.420764 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:22.420765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:22.420778 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:22.420848 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:22.421158 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.421653 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.422379 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.422578 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.422659 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:22.425114 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35259: BlockingRecv error: recv got EOF from 127.0.98.2:35259 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 13:15:22 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:22 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:22.564502 9627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:22.559781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:41781)
0926 13:15:22.559939 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:22.559952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:22.560028 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:22.560216 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:22.560217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:22.560392 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:22.560462 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:22.560464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.561110 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.561111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:22.561657 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:22.561658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.562023 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.562024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:22.562080 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:22.562586 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:22.562591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:22.564445 (+ 1854us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
W20260926 13:15:22.564589 9625 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:22.559892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41781 (local address 127.0.98.62:34135)
0926 13:15:22.559949 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:22.559952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:22.560008 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:15:22.560104 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:22.560105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:22.560119 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:22.560186 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:22.560534 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.561031 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.561741 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:22.561933 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:22.562021 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:22.564526 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41781: BlockingRecv error: recv got EOF from 127.0.98.3:41781 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 13:15:23 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:23 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:23.328222 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:23.316297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:38531)
0926 13:15:23.316431 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:23.316441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:23.316489 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:23.316637 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:23.316638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:23.316807 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:23.316856 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:23.316858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.317519 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.317520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:23.318047 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:23.318049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.318457 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.318459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:23.318503 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:23.325929 (+ 7426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:23.325937 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:23.328159 (+ 2222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
W20260926 13:15:23.328591 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:23.316375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38531 (local address 127.0.98.62:34135)
0926 13:15:23.316491 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:23.316493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:23.316535 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:15:23.316557 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:23.316558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:23.316568 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:23.316619 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:23.316919 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.317432 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.318142 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.318389 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.318454 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:23.328528 (+ 10074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38531: BlockingRecv error: recv got EOF from 127.0.98.1:38531 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
Sep 26 13:15:23 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:23 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:23.430271 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:23.425546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:60819)
0926 13:15:23.425710 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:23.425724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:23.425800 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:23.425974 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:23.425975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:23.426148 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:23.426240 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:23.426242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.426930 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.426931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:23.427481 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:23.427483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.427860 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.427861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:23.427919 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:23.428378 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:23.428383 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:23.430142 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
W20260926 13:15:23.430322 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:23.425664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60819 (local address 127.0.98.62:34135)
0926 13:15:23.425730 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:23.425732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:23.425776 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:15:23.425875 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:23.425875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:23.425889 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:23.425952 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:23.426322 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.426819 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.427570 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.427764 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.427860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:23.430256 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60819: BlockingRecv error: recv got EOF from 127.0.98.2:60819 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 13:15:23 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:23 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:23.569602 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:23.564995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:38809)
0926 13:15:23.565156 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:23.565167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:23.565257 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:23.565412 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:23.565413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:23.565594 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:23.565638 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:23.565640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.566358 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.566360 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:23.566895 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:23.566896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.567254 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.567255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:23.567312 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:23.567819 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:23.567824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:23.569494 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":91,"threads_started":1}
W20260926 13:15:23.569636 9629 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:23.565065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38809 (local address 127.0.98.62:34135)
0926 13:15:23.565127 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:23.565129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:23.565229 (+ 100us) server_negotiation.cc:407] Connection header received
0926 13:15:23.565319 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:23.565321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:23.565336 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:23.565403 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:23.565717 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.566278 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.566987 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:23.567176 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:23.567254 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:23.569571 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38809: BlockingRecv error: recv got EOF from 127.0.98.3:38809 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 13:15:24 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:24 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:24.339895 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:24.328738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:47305)
0926 13:15:24.328889 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:24.328902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:24.328960 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:24.329123 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:24.329124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:24.329281 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:24.329336 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:24.329337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.329963 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.329964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:24.330508 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:24.330510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.330826 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.330827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:24.330866 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:24.337890 (+ 7024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:24.337896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:24.339820 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260926 13:15:24.340001 9633 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:24.328842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47305 (local address 127.0.98.62:34135)
0926 13:15:24.329000 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:24.329002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:24.329010 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:15:24.329034 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:24.329035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:24.329046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:24.329104 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:24.329410 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.329901 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.330575 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.330765 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.330818 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:24.339902 (+ 9084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47305: BlockingRecv error: recv got EOF from 127.0.98.1:47305 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":106,"threads_started":1}
Sep 26 13:15:24 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:24 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:24.435595 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:24.430802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:41701)
0926 13:15:24.430956 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:24.430970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:24.431049 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:24.431184 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:24.431185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:24.431375 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:24.431423 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:24.431425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.432095 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.432096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:24.432644 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:24.432646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.433018 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.433019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:24.433078 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:24.433554 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:24.433559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:24.435501 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260926 13:15:24.435659 9633 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:24.430897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41701 (local address 127.0.98.62:34135)
0926 13:15:24.430971 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:24.430973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:24.431039 (+ 66us) server_negotiation.cc:407] Connection header received
0926 13:15:24.431109 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:24.431110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:24.431121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:24.431179 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:24.431503 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.432008 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.432729 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.432936 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.433017 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:24.435595 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41701: BlockingRecv error: recv got EOF from 127.0.98.2:41701 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 13:15:24 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:24 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:24.574879 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:24.570210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.3:53605)
0926 13:15:24.570359 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:24.570375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:24.570434 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:24.570655 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:24.570656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:24.570909 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:24.570971 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:24.570974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.571704 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.571705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:24.572220 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:24.572222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.572538 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.572539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:24.572576 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:24.573077 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:24.573082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:24.574820 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
W20260926 13:15:24.575042 9633 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:24.570282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53605 (local address 127.0.98.62:34135)
0926 13:15:24.570350 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:24.570352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:24.570418 (+ 66us) server_negotiation.cc:407] Connection header received
0926 13:15:24.570528 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:24.570528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:24.570539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:24.570622 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:24.571092 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.571632 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.572292 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:24.572477 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:24.572529 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:24.574943 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53605: BlockingRecv error: recv got EOF from 127.0.98.3:53605 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 13:15:25 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:25 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:25.345458 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.340448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.1:40317)
0926 13:15:25.340614 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:25.340626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:25.340697 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:25.340835 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:25.340836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:25.341026 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:25.341078 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.341080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.341733 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.341734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.342565 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.342569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.342904 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.342905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.342947 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.343379 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:25.343384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:25.345398 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
W20260926 13:15:25.345564 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:25.340527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40317 (local address 127.0.98.62:34135)
0926 13:15:25.340643 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:25.340644 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:25.340679 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:15:25.340756 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:25.340757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:25.340768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:25.340817 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:25.341144 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.341661 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.342640 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.342831 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.342892 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.345487 (+ 2595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40317: BlockingRecv error: recv got EOF from 127.0.98.1:40317 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
Sep 26 13:15:25 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 13:15:25 dist-test-slave-f864 krb5kdc[8578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 13:15:25.440858 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.436181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34135 (local address 127.0.98.2:47051)
0926 13:15:25.436336 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:25.436346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:25.436394 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:25.436573 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:25.436574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:25.436773 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:25.436825 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.436827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.437463 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.437464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.438008 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.438010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.438333 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.438333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.438377 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.438807 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:25.438812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:25.440744 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:34135: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
W20260926 13:15:25.440965 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:25.436257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47051 (local address 127.0.98.62:34135)
0926 13:15:25.436366 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:25.436367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:25.436380 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:15:25.436456 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:25.436457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:25.436469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:25.436538 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:25.436894 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.437390 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.438076 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.438284 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.438331 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.440871 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47051: BlockingRecv error: recv got EOF from 127.0.98.2:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 13:15:25.494704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9161
I20260926 13:15:25.499117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9309
I20260926 13:15:25.503552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9447
I20260926 13:15:25.507988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9091
2026-09-26T13:15:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17534 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T13:15:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:25Z Disabled control of system clock
I20260926 13:15:25.534731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43007
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 13:15:25.615566 9642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:25.615764 9642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:25.615783 9642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:25.616873 9642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:25.616905 9642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:25.616918 9642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:25.616930 9642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:25.616942 9642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:25.618258 9642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43007
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:25.618491 9642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:25.618729 9642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:25.623870 9647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:25.624001 9642 server_base.cc:1034] running on GCE node
W20260926 13:15:25.623885 9648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:25.623903 9650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:25.624378 9642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:25.624661 9642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:25.625801 9642 hybrid_clock.cc:648] HybridClock initialized: now 1790428525625788 us; error 29 us; skew 500 ppm
I20260926 13:15:25.627144 9642 webserver.cc:460] Webserver started at http://127.0.98.62:32971/ using document root <none> and password file <none>
I20260926 13:15:25.627416 9642 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:25.627487 9642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:25.627635 9642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:25.628739 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "b4f0b35b37fc44ea824887a50b814af8"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "a064c21502b1a65d06505c2c505fd7a6"
server_key_iv: "79de971786653e97b37b44b505db07ae"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.629153 9642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4f0b35b37fc44ea824887a50b814af8"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "a064c21502b1a65d06505c2c505fd7a6"
server_key_iv: "79de971786653e97b37b44b505db07ae"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.630496 9642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:15:25.631265 9656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.631443 9642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:25.631542 9642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "b4f0b35b37fc44ea824887a50b814af8"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "a064c21502b1a65d06505c2c505fd7a6"
server_key_iv: "79de971786653e97b37b44b505db07ae"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.631613 9642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:25.650024 9642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:25.650465 9642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:25.650627 9642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:25.654702 9642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43007
I20260926 13:15:25.654749 9708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43007 every 8 connection(s)
I20260926 13:15:25.655166 9642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:15:25.655885 9709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:25.658551 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8: Bootstrap starting.
I20260926 13:15:25.659082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9642
I20260926 13:15:25.659171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:15:25.659150 9709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:25.659350 392 external_mini_cluster.cc:1428] Setting key 8a4ee83f289b8c772c7a76067a75fd8c
I20260926 13:15:25.659480 9709 log.cc:824] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:25.660254 9709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8: No bootstrap required, opened a new log
I20260926 13:15:25.662415 9709 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f0b35b37fc44ea824887a50b814af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 13:15:25.662588 9709 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:25.662619 9709 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f0b35b37fc44ea824887a50b814af8, State: Initialized, Role: FOLLOWER
I20260926 13:15:25.662741 9709 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f0b35b37fc44ea824887a50b814af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 13:15:25.662832 9709 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:15:25.662869 9709 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:15:25.662914 9709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:25.663491 9709 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f0b35b37fc44ea824887a50b814af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 13:15:25.663601 9709 leader_election.cc:302] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f0b35b37fc44ea824887a50b814af8; no voters:
I20260926 13:15:25.663775 9709 leader_election.cc:288] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:15:25.663831 9714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:25.664000 9714 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [term 1 LEADER]: Becoming Leader. State: Replica: b4f0b35b37fc44ea824887a50b814af8, State: Running, Role: LEADER
I20260926 13:15:25.664103 9709 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:15:25.664152 9714 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f0b35b37fc44ea824887a50b814af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } }
I20260926 13:15:25.664551 9715 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f0b35b37fc44ea824887a50b814af8. Latest consensus state: current_term: 1 leader_uuid: "b4f0b35b37fc44ea824887a50b814af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f0b35b37fc44ea824887a50b814af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } } }
I20260926 13:15:25.664778 9715 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:25.665124 9723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:15:25.665621 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.659847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.0.98.62:43007)
0926 13:15:25.660026 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:25.660033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:25.660057 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:15:25.660510 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:25.660517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:25.660575 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:25.660699 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:25.660959 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.661842 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.663010 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.663432 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.663526 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.664286 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:25.664290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:25.664290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:25.664514 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:25.664556 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:25.664571 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:25.664642 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:25.664942 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 13:15:25.665038 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260926 13:15:25.665778 9716 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f0b35b37fc44ea824887a50b814af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f0b35b37fc44ea824887a50b814af8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43007 } } }
I20260926 13:15:25.665939 9716 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:25.666276 9723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:15:25.667825 9723 catalog_manager.cc:1347] Generated new cluster ID: 67a741b365a644749e792c3e6d33911e
I20260926 13:15:25.667876 9723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:15:25.680120 9723 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:15:25.680625 9723 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:15:25.690002 9723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f0b35b37fc44ea824887a50b814af8: Generated new TSK 0
I20260926 13:15:25.690254 9723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:15:25.692780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 13:15:25.775038 9733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:25.775255 9733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:25.775277 9733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:25.776477 9733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:25.776521 9733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:25.776548 9733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:15:25.777916 9733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:25.778195 9733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:25.778506 9733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:25.783425 9738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:25.783425 9739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:25.783663 9733 server_base.cc:1034] running on GCE node
W20260926 13:15:25.783444 9741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:25.783926 9733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:25.784174 9733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:25.785346 9733 hybrid_clock.cc:648] HybridClock initialized: now 1790428525785331 us; error 36 us; skew 500 ppm
I20260926 13:15:25.786674 9733 webserver.cc:460] Webserver started at http://127.0.98.1:44643/ using document root <none> and password file <none>
I20260926 13:15:25.786947 9733 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:25.787019 9733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:25.787158 9733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:25.788259 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "f9d49d33c37f4dec8aa8ce5cb1799562"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "60f6bfafd4f2cdf4ca009312a0946e75"
server_key_iv: "3015632202855974429f9b8287dd2abf"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.788625 9733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f9d49d33c37f4dec8aa8ce5cb1799562"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "60f6bfafd4f2cdf4ca009312a0946e75"
server_key_iv: "3015632202855974429f9b8287dd2abf"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.789988 9733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:15:25.790865 9747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.791038 9733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:25.791131 9733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "f9d49d33c37f4dec8aa8ce5cb1799562"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "60f6bfafd4f2cdf4ca009312a0946e75"
server_key_iv: "3015632202855974429f9b8287dd2abf"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.791214 9733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:25.804934 9733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:25.805352 9733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:25.805518 9733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:25.805760 9733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:25.806149 9733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:25.806254 9733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.806304 9733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:25.806332 9733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.812119 9733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42601
I20260926 13:15:25.812181 9860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42601 every 8 connection(s)
I20260926 13:15:25.812554 9733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:15:25.817509 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.813326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54557 (local address 127.0.98.62:43007)
0926 13:15:25.813373 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:25.813375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:25.813609 (+ 234us) server_negotiation.cc:407] Connection header received
0926 13:15:25.814098 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:25.814099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:25.814111 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:25.814229 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:25.814880 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.815369 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.816317 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.816495 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.816545 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.817287 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:25.817289 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:25.817289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:25.817293 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:25.817307 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:25.817310 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:25.817352 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:25.817429 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:15:25.817449 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:15:25.817723 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.813285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43007 (local address 127.0.98.1:54557)
0926 13:15:25.813479 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:25.813521 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:25.814018 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:25.814298 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:25.814302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:25.814310 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:25.814800 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.814803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.815445 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.815446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.816245 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.816247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.816561 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.816562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.816666 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.817163 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:15:25.817164 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:25.817192 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:15:25.817214 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:25.817363 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:25.817367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:25.817369 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:25.817423 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:15:25.817511 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260926 13:15:25.817988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9733
I20260926 13:15:25.818086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:15:25.818261 9861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43007
I20260926 13:15:25.818341 9861 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:25.818367 392 external_mini_cluster.cc:1428] Setting key 4adc9585fed8e7dee02ab9388abe445f
I20260926 13:15:25.818560 9861 heartbeater.cc:511] Master 127.0.98.62:43007 requested a full tablet report, sending...
I20260926 13:15:25.819078 9672 ts_manager.cc:194] Registered new tserver with Master: f9d49d33c37f4dec8aa8ce5cb1799562 (127.0.98.1:42601)
I20260926 13:15:25.819329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 13:15:25.819671 9672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:54557
W20260926 13:15:25.898947 9864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:25.899171 9864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:25.899195 9864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:25.900336 9864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:25.900372 9864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:25.900396 9864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:15:25.901795 9864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:25.901995 9864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:25.902318 9864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:25.907385 9872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:25.907337 9869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:25.907344 9870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:25.907410 9864 server_base.cc:1034] running on GCE node
I20260926 13:15:25.907745 9864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:25.908025 9864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:25.909183 9864 hybrid_clock.cc:648] HybridClock initialized: now 1790428525909171 us; error 40 us; skew 500 ppm
I20260926 13:15:25.910614 9864 webserver.cc:460] Webserver started at http://127.0.98.2:37615/ using document root <none> and password file <none>
I20260926 13:15:25.910885 9864 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:25.910957 9864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:25.911096 9864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:25.912189 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "6bec62490f364d0381b5dc8c5555dbe4"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "affe100b4be6820c627abbcbeb114cd5"
server_key_iv: "c368fe7384f67bee44a34bbcb092cde1"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.912578 9864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6bec62490f364d0381b5dc8c5555dbe4"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "affe100b4be6820c627abbcbeb114cd5"
server_key_iv: "c368fe7384f67bee44a34bbcb092cde1"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.913872 9864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:15:25.914630 9878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.914788 9864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.914885 9864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "6bec62490f364d0381b5dc8c5555dbe4"
format_stamp: "Formatted at 2026-09-26 13:15:25 on dist-test-slave-f864"
server_key: "affe100b4be6820c627abbcbeb114cd5"
server_key_iv: "c368fe7384f67bee44a34bbcb092cde1"
server_key_version: "encryptionkey@0"
I20260926 13:15:25.914958 9864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:25.951000 9864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:25.951344 9864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:25.951499 9864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:25.951738 9864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:25.952080 9864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:25.952129 9864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.952170 9864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:25.952198 9864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:25.957857 9864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43383
I20260926 13:15:25.957906 9991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43383 every 8 connection(s)
I20260926 13:15:25.958267 9864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:15:25.963250 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.958990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51561 (local address 127.0.98.62:43007)
0926 13:15:25.959032 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:25.959034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:25.959289 (+ 255us) server_negotiation.cc:407] Connection header received
0926 13:15:25.959763 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:25.959764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:25.959774 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:25.959838 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:25.960475 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.960964 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.961932 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.962110 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.962181 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.963047 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:25.963048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:25.963048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:25.963052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:25.963062 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:25.963066 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:25.963110 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:25.963171 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 13:15:25.963191 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:15:25.963500 9994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:25.958995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43007 (local address 127.0.98.2:51561)
0926 13:15:25.959169 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:25.959204 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:25.959681 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:25.959881 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:25.959884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:25.959892 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:25.960406 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.960409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.961048 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.961049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.961858 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:25.961860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:25.962228 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:25.962229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:25.962356 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:25.962925 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:15:25.962925 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:25.962955 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:15:25.962978 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:25.963112 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:25.963115 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:25.963117 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:25.963164 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:15:25.963250 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260926 13:15:25.963991 9992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43007
I20260926 13:15:25.964092 9992 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:25.964241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9864
I20260926 13:15:25.964282 9992 heartbeater.cc:511] Master 127.0.98.62:43007 requested a full tablet report, sending...
I20260926 13:15:25.964373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:15:25.964558 392 external_mini_cluster.cc:1428] Setting key 85d43a2161cca826485091e1c13b66ff
I20260926 13:15:25.964673 9672 ts_manager.cc:194] Registered new tserver with Master: 6bec62490f364d0381b5dc8c5555dbe4 (127.0.98.2:43383)
I20260926 13:15:25.965096 9672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:51561
I20260926 13:15:25.965528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--builtin_ntp_servers=127.0.98.20:33427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 13:15:26.046345 9995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:26.046567 9995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:26.046589 9995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:26.047785 9995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:26.047833 9995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:26.047859 9995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:15:26.049198 9995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:26.049461 9995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:26.049762 9995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:26.054811 10001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.054864 10003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.054944 9995 server_base.cc:1034] running on GCE node
W20260926 13:15:26.054812 10000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.055259 9995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:26.055459 9995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:26.056592 9995 hybrid_clock.cc:648] HybridClock initialized: now 1790428526056566 us; error 44 us; skew 500 ppm
I20260926 13:15:26.057723 9995 webserver.cc:460] Webserver started at http://127.0.98.3:43979/ using document root <none> and password file <none>
I20260926 13:15:26.057924 9995 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:26.057963 9995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:26.058053 9995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:26.059127 9995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "3baad20187274105a671a69792f3c2b9"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "e040e1a5b7ce71abda5a9ddd59bc0cca"
server_key_iv: "e67d1a25d8073336fa76b0ddcd48d691"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.059432 9995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3baad20187274105a671a69792f3c2b9"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "e040e1a5b7ce71abda5a9ddd59bc0cca"
server_key_iv: "e67d1a25d8073336fa76b0ddcd48d691"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.060745 9995 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:15:26.061625 10009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.061842 9995 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:26.061928 9995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "3baad20187274105a671a69792f3c2b9"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "e040e1a5b7ce71abda5a9ddd59bc0cca"
server_key_iv: "e67d1a25d8073336fa76b0ddcd48d691"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.061988 9995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:26.078640 9995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:26.078918 9995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:26.079020 9995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:26.079221 9995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:26.079509 9995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:26.079538 9995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.079561 9995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:26.079578 9995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.085043 9995 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39457
I20260926 13:15:26.085146 10122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39457 every 8 connection(s)
I20260926 13:15:26.085453 9995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:15:26.090092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 9995
I20260926 13:15:26.090253 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:15:26.090483 392 external_mini_cluster.cc:1428] Setting key ca6acb8f9de45b81f070b7f7739626e0
I20260926 13:15:26.091002 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.086278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36863 (local address 127.0.98.62:43007)
0926 13:15:26.086362 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.086364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.086573 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:15:26.087119 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.087120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.087132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.087196 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.087921 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:26.088411 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:26.089399 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:26.089596 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:26.089669 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:26.090650 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:26.090652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:26.090652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:26.090658 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:26.090677 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:26.090681 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:26.090732 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:26.090879 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 13:15:26.090907 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:15:26.091250 10125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.086228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43007 (local address 127.0.98.3:36863)
0926 13:15:26.086452 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:26.086489 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:26.087011 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:26.087270 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:26.087274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:26.087289 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:26.087819 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:26.087825 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:26.088512 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:26.088513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:26.089300 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:26.089302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:26.089686 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:26.089687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:26.089823 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:26.090478 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:15:26.090479 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:26.090518 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:15:26.090549 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:26.090783 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:26.090790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:26.090792 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:26.090888 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 13:15:26.091006 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260926 13:15:26.091670 10123 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43007
I20260926 13:15:26.091763 10123 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:26.091905 10123 heartbeater.cc:511] Master 127.0.98.62:43007 requested a full tablet report, sending...
I20260926 13:15:26.092235 9672 ts_manager.cc:194] Registered new tserver with Master: 3baad20187274105a671a69792f3c2b9 (127.0.98.3:39457)
I20260926 13:15:26.092566 9672 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36863
I20260926 13:15:26.102386 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:15:26.104161 9713 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:26.103686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40456 (local address 127.0.98.62:43007)
0926 13:15:26.103748 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.103750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.103854 (+ 104us) server_negotiation.cc:407] Connection header received
0926 13:15:26.103900 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.103901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.103910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.103951 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.104084 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40456: BlockingRecv error: recv got EOF from 127.0.0.1:40456 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:15:26.104208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9733
I20260926 13:15:26.108695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9864
I20260926 13:15:26.112785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9995
I20260926 13:15:26.116788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 9642
2026-09-26T13:15:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (607 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T13:15:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:26Z Disabled control of system clock
I20260926 13:15:26.142602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38347
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:15:26.221563 10134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:26.221774 10134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:26.221841 10134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:26.223527 10134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:26.223574 10134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:26.223587 10134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:26.223600 10134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:26.223613 10134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:26.224897 10134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:26.225087 10134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:26.225319 10134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:26.230186 10142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.230213 10139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.230225 10140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.230477 10134 server_base.cc:1034] running on GCE node
I20260926 13:15:26.230727 10134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:26.231015 10134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:26.232187 10134 hybrid_clock.cc:648] HybridClock initialized: now 1790428526232168 us; error 45 us; skew 500 ppm
I20260926 13:15:26.233397 10134 webserver.cc:460] Webserver started at http://127.0.98.62:41333/ using document root <none> and password file <none>
I20260926 13:15:26.233630 10134 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:26.233693 10134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:26.233850 10134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:26.234992 10134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "52bc834ebb514aeeb6d12cc400f5b653"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "db183f2b1e8e216d8b59e788892d1365"
server_key_iv: "7ac9f6864266144df9c0a609b1d3f645"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.235354 10134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "52bc834ebb514aeeb6d12cc400f5b653"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "db183f2b1e8e216d8b59e788892d1365"
server_key_iv: "7ac9f6864266144df9c0a609b1d3f645"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.236665 10134 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:15:26.237316 10148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.237470 10134 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:15:26.237565 10134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "52bc834ebb514aeeb6d12cc400f5b653"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "db183f2b1e8e216d8b59e788892d1365"
server_key_iv: "7ac9f6864266144df9c0a609b1d3f645"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.237632 10134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:26.240873 10134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:26.241114 10134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:26.241247 10134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:26.244975 10134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38347
I20260926 13:15:26.245003 10200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38347 every 8 connection(s)
I20260926 13:15:26.245326 10134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:15:26.245965 10201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:26.246495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10134
I20260926 13:15:26.246587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:15:26.246794 392 external_mini_cluster.cc:1428] Setting key f132150134a40b47a173cda2a307394f
I20260926 13:15:26.249437 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.247333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58182 (local address 127.0.98.62:38347)
0926 13:15:26.247534 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.247543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.247574 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:15:26.248070 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.248079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.248135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.248287 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.249027 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:26.249030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:26.249030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:26.249035 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:26.249079 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:26.249092 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:26.249171 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:26.249208 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:15:26.249260 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260926 13:15:26.249854 10201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653: Bootstrap starting.
I20260926 13:15:26.250535 10201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:26.250748 10201 log.cc:824] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:26.251361 10201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653: No bootstrap required, opened a new log
I20260926 13:15:26.252590 10201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc834ebb514aeeb6d12cc400f5b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38347 } }
I20260926 13:15:26.252715 10201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:26.252737 10201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bc834ebb514aeeb6d12cc400f5b653, State: Initialized, Role: FOLLOWER
I20260926 13:15:26.252851 10201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc834ebb514aeeb6d12cc400f5b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38347 } }
I20260926 13:15:26.252933 10201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:15:26.252955 10201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:15:26.252983 10201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:26.253459 10201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc834ebb514aeeb6d12cc400f5b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38347 } }
I20260926 13:15:26.253539 10201 leader_election.cc:302] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52bc834ebb514aeeb6d12cc400f5b653; no voters:
I20260926 13:15:26.253649 10201 leader_election.cc:288] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:15:26.253875 10201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:15:26.253969 10206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:26.254097 10206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [term 1 LEADER]: Becoming Leader. State: Replica: 52bc834ebb514aeeb6d12cc400f5b653, State: Running, Role: LEADER
I20260926 13:15:26.254258 10206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc834ebb514aeeb6d12cc400f5b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38347 } }
I20260926 13:15:26.254730 10207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52bc834ebb514aeeb6d12cc400f5b653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc834ebb514aeeb6d12cc400f5b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38347 } } }
I20260926 13:15:26.254746 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bc834ebb514aeeb6d12cc400f5b653. Latest consensus state: current_term: 1 leader_uuid: "52bc834ebb514aeeb6d12cc400f5b653" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bc834ebb514aeeb6d12cc400f5b653" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38347 } } }
I20260926 13:15:26.254846 10207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:26.254856 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:26.255064 10217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:15:26.255537 10217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:15:26.256860 10217 catalog_manager.cc:1347] Generated new cluster ID: cfde2a4b56d3481bb2bf594f2da7b21b
I20260926 13:15:26.256913 10217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:15:26.279279 10217 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:15:26.279474 10217 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:15:26.286870 10217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52bc834ebb514aeeb6d12cc400f5b653: Generated new TSK 0
I20260926 13:15:26.287091 10217 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:15:26.299270 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38347
--builtin_ntp_servers=127.0.98.20:37375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:15:26.381163 10225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:26.381356 10225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:26.381377 10225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:26.382577 10225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:26.382623 10225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:26.382656 10225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:15:26.384001 10225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:26.384253 10225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:26.384552 10225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:26.391158 10230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.391225 10231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.391487 10233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.392733 10225 server_base.cc:1034] running on GCE node
I20260926 13:15:26.393018 10225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:26.393301 10225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:26.394466 10225 hybrid_clock.cc:648] HybridClock initialized: now 1790428526394438 us; error 43 us; skew 500 ppm
I20260926 13:15:26.396133 10225 webserver.cc:460] Webserver started at http://127.0.98.1:39777/ using document root <none> and password file <none>
I20260926 13:15:26.396462 10225 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:26.396546 10225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:26.396821 10225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:26.398514 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "17272b9ba69b414ebdb5496180d00e59"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "0ac64d8493571de25f458a71dc11b1da"
server_key_iv: "f2c1299c1e7babbbe3f43a4c10455c96"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.398972 10225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "17272b9ba69b414ebdb5496180d00e59"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "0ac64d8493571de25f458a71dc11b1da"
server_key_iv: "f2c1299c1e7babbbe3f43a4c10455c96"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.400748 10225 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:15:26.401930 10239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.402109 10225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:26.402314 10225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "17272b9ba69b414ebdb5496180d00e59"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "0ac64d8493571de25f458a71dc11b1da"
server_key_iv: "f2c1299c1e7babbbe3f43a4c10455c96"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.402400 10225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:26.406354 10225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:26.406672 10225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:26.406829 10225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:26.407073 10225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:26.407446 10225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:26.407503 10225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.407539 10225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:26.407563 10225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.415017 10225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44931
I20260926 13:15:26.415060 10352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44931 every 8 connection(s)
I20260926 13:15:26.415385 10225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:15:26.418248 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.416061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34033 (local address 127.0.98.62:38347)
0926 13:15:26.416119 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.416121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.416343 (+ 222us) server_negotiation.cc:407] Connection header received
0926 13:15:26.416845 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.416846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.416853 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.416904 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.417994 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:26.417996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:26.417996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:26.418001 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:26.418014 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:26.418019 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:26.418069 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:26.418144 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:15:26.418177 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:15:26.418447 10355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.416048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38347 (local address 127.0.98.1:34033)
0926 13:15:26.416232 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:26.416264 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:26.416738 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:26.416973 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:26.416977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:26.416987 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:26.417856 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:15:26.417856 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:26.417883 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:15:26.417910 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:26.418067 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:26.418072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:26.418075 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:15:26.418155 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 13:15:26.418235 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260926 13:15:26.418901 10353 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38347
I20260926 13:15:26.418992 10353 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:26.419139 10353 heartbeater.cc:511] Master 127.0.98.62:38347 requested a full tablet report, sending...
I20260926 13:15:26.419618 10165 ts_manager.cc:194] Registered new tserver with Master: 17272b9ba69b414ebdb5496180d00e59 (127.0.98.1:44931)
I20260926 13:15:26.424504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10225
I20260926 13:15:26.424576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:15:26.424772 392 external_mini_cluster.cc:1428] Setting key 20ec67aeb97d37c8756fa05bf63b9bf0
I20260926 13:15:26.425727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38347
--builtin_ntp_servers=127.0.98.20:37375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:15:26.506399 10356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:26.506595 10356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:26.506618 10356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:26.507709 10356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:26.507742 10356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:26.507766 10356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:15:26.509001 10356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:26.509181 10356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:26.509411 10356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:26.514312 10362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.514290 10361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.514290 10364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.514539 10356 server_base.cc:1034] running on GCE node
I20260926 13:15:26.514782 10356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:26.515038 10356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:26.516196 10356 hybrid_clock.cc:648] HybridClock initialized: now 1790428526516186 us; error 31 us; skew 500 ppm
I20260926 13:15:26.517424 10356 webserver.cc:460] Webserver started at http://127.0.98.2:43041/ using document root <none> and password file <none>
I20260926 13:15:26.517663 10356 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:26.517750 10356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:26.517885 10356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:26.519001 10356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "ac8bae4b5c86494a84ac01adf8837a5f"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "5a11e7425ea81c87f0323e388a7f04a5"
server_key_iv: "212ab3e868992e93f6dcf52d65828da7"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.519361 10356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ac8bae4b5c86494a84ac01adf8837a5f"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "5a11e7425ea81c87f0323e388a7f04a5"
server_key_iv: "212ab3e868992e93f6dcf52d65828da7"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.520681 10356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:15:26.521371 10370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.521536 10356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:15:26.521632 10356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "ac8bae4b5c86494a84ac01adf8837a5f"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "5a11e7425ea81c87f0323e388a7f04a5"
server_key_iv: "212ab3e868992e93f6dcf52d65828da7"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.521721 10356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:26.525058 10356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:26.525314 10356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:26.525451 10356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:26.525653 10356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:26.525959 10356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:26.526010 10356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.526049 10356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:26.526077 10356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.531945 10356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40459
I20260926 13:15:26.531982 10483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40459 every 8 connection(s)
I20260926 13:15:26.532332 10356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:15:26.535298 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.533054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55191 (local address 127.0.98.62:38347)
0926 13:15:26.533113 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.533115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.533359 (+ 244us) server_negotiation.cc:407] Connection header received
0926 13:15:26.533825 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.533826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.533832 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.533892 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.535088 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:26.535089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:26.535089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:26.535093 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:26.535107 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:26.535111 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:26.535160 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:26.535228 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:15:26.535241 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:15:26.535451 10486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.533059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38347 (local address 127.0.98.2:55191)
0926 13:15:26.533228 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:26.533260 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:26.533720 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:26.533964 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:26.533967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:26.533975 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:26.534942 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:15:26.534943 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:26.534971 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:15:26.535000 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:26.535157 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:26.535160 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:26.535162 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:26.535212 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:15:26.535259 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260926 13:15:26.535949 10484 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38347
I20260926 13:15:26.536043 10484 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:26.536185 10484 heartbeater.cc:511] Master 127.0.98.62:38347 requested a full tablet report, sending...
I20260926 13:15:26.536541 10165 ts_manager.cc:194] Registered new tserver with Master: ac8bae4b5c86494a84ac01adf8837a5f (127.0.98.2:40459)
I20260926 13:15:26.540136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10356
I20260926 13:15:26.540261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:15:26.540457 392 external_mini_cluster.cc:1428] Setting key 703bcd68748236adda181412a0552e8f
I20260926 13:15:26.541353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38347
--builtin_ntp_servers=127.0.98.20:37375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 13:15:26.622534 10487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:26.622773 10487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:26.622812 10487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:26.623956 10487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:26.624020 10487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:26.624085 10487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:15:26.625453 10487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:26.625687 10487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:26.625968 10487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:26.630897 10492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.630985 10487 server_base.cc:1034] running on GCE node
W20260926 13:15:26.630929 10493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:26.630911 10495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:26.631366 10487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:26.631611 10487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:26.632778 10487 hybrid_clock.cc:648] HybridClock initialized: now 1790428526632763 us; error 30 us; skew 500 ppm
I20260926 13:15:26.634068 10487 webserver.cc:460] Webserver started at http://127.0.98.3:42079/ using document root <none> and password file <none>
I20260926 13:15:26.634351 10487 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:26.634435 10487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:26.634577 10487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:26.635785 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "df856beec4ab46d0b08192429127111a"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "82de09ec825278c84cc4b92ac0b54f5f"
server_key_iv: "57e1e28a3faa927060991d13e74c5a04"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.636160 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "df856beec4ab46d0b08192429127111a"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "82de09ec825278c84cc4b92ac0b54f5f"
server_key_iv: "57e1e28a3faa927060991d13e74c5a04"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.637481 10487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:15:26.638245 10501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.638402 10487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:26.638496 10487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "df856beec4ab46d0b08192429127111a"
format_stamp: "Formatted at 2026-09-26 13:15:26 on dist-test-slave-f864"
server_key: "82de09ec825278c84cc4b92ac0b54f5f"
server_key_iv: "57e1e28a3faa927060991d13e74c5a04"
server_key_version: "encryptionkey@0"
I20260926 13:15:26.638578 10487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:26.642046 10487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:26.642341 10487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:26.642480 10487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:26.642684 10487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:26.642995 10487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:26.643044 10487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.643083 10487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:26.643110 10487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:26.648581 10487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34909
I20260926 13:15:26.648646 10614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34909 every 8 connection(s)
I20260926 13:15:26.648977 10487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:15:26.651937 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.649668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46733 (local address 127.0.98.62:38347)
0926 13:15:26.649727 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.649729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.649936 (+ 207us) server_negotiation.cc:407] Connection header received
0926 13:15:26.650485 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.650486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.650491 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.650544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.651713 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:26.651715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:26.651715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:15:26.651719 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:26.651734 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:15:26.651738 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:26.651794 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:26.651871 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:15:26.651883 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:15:26.652107 10617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:26.649676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38347 (local address 127.0.98.3:46733)
0926 13:15:26.649834 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:26.649866 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:26.650392 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:26.650597 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:26.650600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:26.650609 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:26.651551 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:15:26.651551 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:26.651580 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:15:26.651609 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:26.651804 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:26.651808 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:26.651810 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:15:26.651868 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:15:26.651916 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260926 13:15:26.652567 10615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38347
I20260926 13:15:26.652662 10615 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:26.652805 10615 heartbeater.cc:511] Master 127.0.98.62:38347 requested a full tablet report, sending...
I20260926 13:15:26.653151 10165 ts_manager.cc:194] Registered new tserver with Master: df856beec4ab46d0b08192429127111a (127.0.98.3:34909)
I20260926 13:15:26.655717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10487
I20260926 13:15:26.655826 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:15:26.656013 392 external_mini_cluster.cc:1428] Setting key a8f423c6a87852e266ee9300ea9f6575
I20260926 13:15:26.657166 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:15:26.659216 10204 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:15:26.658802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.62:38347)
0926 13:15:26.658863 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:26.658865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:26.658922 (+ 57us) server_negotiation.cc:407] Connection header received
0926 13:15:26.658990 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:26.658990 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:26.658995 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:26.659052 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:26.659162 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36420: BlockingRecv error: recv got EOF from 127.0.0.1:36420 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:15:26.659368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10225
I20260926 13:15:26.663869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10356
I20260926 13:15:26.669044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10487
I20260926 13:15:26.673545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10134
2026-09-26T13:15:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (557 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:15:26 dist-test-slave-f864 krb5kdc[10625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:15:26 dist-test-slave-f864 krb5kdc[10625](info): set up 2 sockets
Sep 26 13:15:26 dist-test-slave-f864 krb5kdc[10625](info): commencing operation
krb5kdc: starting...
W20260926 13:15:27.722476 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:15:27 dist-test-slave-f864 krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:15:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:15:27.845759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43385
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:27.927222 10638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:27.927487 10638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:27.927526 10638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:27.928730 10638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:15:27.928807 10638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:27.928829 10638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:27.928849 10638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:15:27.928903 10638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:15:27.930437 10638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:27.930729 10638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:27.930960 10638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:27.936028 10643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:27.936124 10638 server_base.cc:1034] running on GCE node
W20260926 13:15:27.936035 10644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:27.936045 10646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:27.936580 10638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:27.936923 10638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:27.938086 10638 hybrid_clock.cc:648] HybridClock initialized: now 1790428527938062 us; error 42 us; skew 500 ppm
Sep 26 13:15:27 dist-test-slave-f864 krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:27.940479 10638 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:15:27.940910 10638 webserver.cc:460] Webserver started at http://127.0.98.62:41103/ using document root <none> and password file <none>
I20260926 13:15:27.941145 10638 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:27.941210 10638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:27.941355 10638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:27.942488 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "9820bb1bc0b847e09d7720cad1d66685"
format_stamp: "Formatted at 2026-09-26 13:15:27 on dist-test-slave-f864"
server_key: "a1da69077e0641df78a71788d1a3c2ee"
server_key_iv: "dd9de1742b2944b862d0a079cf0f009f"
server_key_version: "encryptionkey@0"
I20260926 13:15:27.942853 10638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "9820bb1bc0b847e09d7720cad1d66685"
format_stamp: "Formatted at 2026-09-26 13:15:27 on dist-test-slave-f864"
server_key: "a1da69077e0641df78a71788d1a3c2ee"
server_key_iv: "dd9de1742b2944b862d0a079cf0f009f"
server_key_version: "encryptionkey@0"
I20260926 13:15:27.944193 10638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:15:27.944947 10653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:27.945174 10638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:15:27.945264 10638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "9820bb1bc0b847e09d7720cad1d66685"
format_stamp: "Formatted at 2026-09-26 13:15:27 on dist-test-slave-f864"
server_key: "a1da69077e0641df78a71788d1a3c2ee"
server_key_iv: "dd9de1742b2944b862d0a079cf0f009f"
server_key_version: "encryptionkey@0"
I20260926 13:15:27.945348 10638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:27.959326 10638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:27.961261 10638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:27.961431 10638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:27.965349 10638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43385
I20260926 13:15:27.965382 10705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43385 every 8 connection(s)
I20260926 13:15:27.965727 10638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:15:27.966847 10706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:27.969080 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685: Bootstrap starting.
I20260926 13:15:27.969620 10706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:27.969848 10706 log.cc:824] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:27.970599 10706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685: No bootstrap required, opened a new log
I20260926 13:15:27.970850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10638
I20260926 13:15:27.970937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:15:27.971118 392 external_mini_cluster.cc:1428] Setting key 8bf0432d542c6bf5528d3da2fb89e8c4
I20260926 13:15:27.972716 10706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9820bb1bc0b847e09d7720cad1d66685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260926 13:15:27.972894 10706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:27.972936 10706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9820bb1bc0b847e09d7720cad1d66685, State: Initialized, Role: FOLLOWER
I20260926 13:15:27.973176 10706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9820bb1bc0b847e09d7720cad1d66685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260926 13:15:27.973277 10706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:15:27.973315 10706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:15:27.973373 10706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:27.974442 10706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9820bb1bc0b847e09d7720cad1d66685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260926 13:15:27.974535 10706 leader_election.cc:302] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9820bb1bc0b847e09d7720cad1d66685; no voters:
I20260926 13:15:27.974741 10706 leader_election.cc:288] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:15:27.974766 10711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:27.974872 10711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [term 1 LEADER]: Becoming Leader. State: Replica: 9820bb1bc0b847e09d7720cad1d66685, State: Running, Role: LEADER
I20260926 13:15:27.974953 10711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9820bb1bc0b847e09d7720cad1d66685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } }
I20260926 13:15:27.975013 10706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:15:27.975265 10711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9820bb1bc0b847e09d7720cad1d66685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9820bb1bc0b847e09d7720cad1d66685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } } }
I20260926 13:15:27.975333 10711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:27.975461 10711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9820bb1bc0b847e09d7720cad1d66685. Latest consensus state: current_term: 1 leader_uuid: "9820bb1bc0b847e09d7720cad1d66685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9820bb1bc0b847e09d7720cad1d66685" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43385 } } }
I20260926 13:15:27.975528 10711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685 [sys.catalog]: This master's current role is: LEADER
I20260926 13:15:27.975662 10716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:15:27.976449 10716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:15:27.978183 10716 catalog_manager.cc:1347] Generated new cluster ID: 5389166022244be9aadba0f06a2890df
I20260926 13:15:27.978233 10716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:15:27 dist-test-slave-f864 krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:27.982774 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:27.971649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33276 (local address 127.0.98.62:43385)
0926 13:15:27.971819 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:27.971836 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:27.971862 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:15:27.972691 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:27.972698 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:27.972748 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:27.972868 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:27.973727 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:27.974815 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:27.975919 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:27.976581 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:27.977078 (+ 497us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:27.979400 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:27.979404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:27.979405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:27.979417 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:27.981125 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:27.981510 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:27.981514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:27.981514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:27.981627 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:27.981847 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:27.981848 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:27.981849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:27.982094 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:27.982307 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:27.982375 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:15:27.982623 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260926 13:15:27.987195 10716 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:15:27.987726 10716 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:15:28.003736 10716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685: Generated new TSK 0
I20260926 13:15:28.003984 10716 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:15:28.037451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:28.118772 10732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:28.119017 10732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:28.119055 10732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:28.120223 10732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:28.120299 10732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:28.120344 10732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:15:28.121794 10732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:28.122064 10732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:28.122340 10732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:15:28.127658 10732 server_base.cc:1034] running on GCE node
W20260926 13:15:28.127638 10738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:28.127638 10737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:28.127638 10740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:28.128127 10732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:28.128440 10732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:28.129655 10732 hybrid_clock.cc:648] HybridClock initialized: now 1790428528129620 us; error 47 us; skew 500 ppm
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:28.132366 10732 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:15:28.132831 10732 webserver.cc:460] Webserver started at http://127.0.98.1:45403/ using document root <none> and password file <none>
I20260926 13:15:28.133071 10732 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:28.133140 10732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:28.133280 10732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:28.134538 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bf840a09ed044f78d5da33de3cb63a1"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "b188cc6a6b28e4dcfcb71a5c5cabed4c"
server_key_iv: "d50a95126a7e1d8443b284074d477c96"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.134915 10732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bf840a09ed044f78d5da33de3cb63a1"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "b188cc6a6b28e4dcfcb71a5c5cabed4c"
server_key_iv: "d50a95126a7e1d8443b284074d477c96"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.136327 10732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 13:15:28.137161 10747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.137348 10732 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:28.137445 10732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "1bf840a09ed044f78d5da33de3cb63a1"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "b188cc6a6b28e4dcfcb71a5c5cabed4c"
server_key_iv: "d50a95126a7e1d8443b284074d477c96"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.137531 10732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:28.156955 10732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:28.158890 10732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:28.159076 10732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:28.159325 10732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:28.159648 10732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:28.159704 10732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.159739 10732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:28.159765 10732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.165551 10732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36175
I20260926 13:15:28.165632 10860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36175 every 8 connection(s)
I20260926 13:15:28.165966 10732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:28.172920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10732
I20260926 13:15:28.173031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:15:28.173249 392 external_mini_cluster.cc:1428] Setting key 9ba2e6404102cef6d69d30767681c766
I20260926 13:15:28.176198 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.166772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54171 (local address 127.0.98.62:43385)
0926 13:15:28.166861 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.166863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.167053 (+ 190us) server_negotiation.cc:407] Connection header received
0926 13:15:28.167542 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.167543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.167555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.167613 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:28.168530 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.169067 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.170124 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.170400 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.170461 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.172688 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:28.172689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:28.172690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:28.172697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:28.174659 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.175092 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.175093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.175094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.175159 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.175452 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.175454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.175454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.175613 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:28.175689 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.176053 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.176131 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 13:15:28.176488 10863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.166740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43385 (local address 127.0.98.1:54171)
0926 13:15:28.166908 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.166952 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.167455 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.167678 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.167681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.168244 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:28.168438 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.168443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.169176 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.169177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.170021 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.170024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.170482 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.170483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.170636 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.171144 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:28.171153 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:28.172599 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:28.174766 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:28.174773 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:28.174776 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:28.175017 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:28.175246 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:28.175247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:28.175247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:28.175368 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:28.175685 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:28.175692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:28.175893 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.176023 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.176216 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260926 13:15:28.177083 10861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43385
I20260926 13:15:28.177192 10861 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:28.177383 10861 heartbeater.cc:511] Master 127.0.98.62:43385 requested a full tablet report, sending...
I20260926 13:15:28.177983 10670 ts_manager.cc:194] Registered new tserver with Master: 1bf840a09ed044f78d5da33de3cb63a1 (127.0.98.1:36175)
I20260926 13:15:28.178992 10670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54171
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:15:28.201483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:28.283998 10866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:28.284199 10866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:28.284224 10866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:28.285311 10866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:28.285347 10866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:28.285372 10866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:15:28.286849 10866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:28.287036 10866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:28.287212 10866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:28.292284 10872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:28.292198 10871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:28.292207 10874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:28.292484 10866 server_base.cc:1034] running on GCE node
I20260926 13:15:28.292786 10866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:28.293023 10866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:28.294250 10866 hybrid_clock.cc:648] HybridClock initialized: now 1790428528294243 us; error 36 us; skew 500 ppm
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:28.296769 10866 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:15:28.297214 10866 webserver.cc:460] Webserver started at http://127.0.98.2:37413/ using document root <none> and password file <none>
I20260926 13:15:28.297437 10866 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:28.297521 10866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:28.297657 10866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:28.298807 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "368b74b9f631d697d5b6c3fc084772d6"
server_key_iv: "c42dbbe3134a079be54a9256e5c5a3b0"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.299180 10866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "368b74b9f631d697d5b6c3fc084772d6"
server_key_iv: "c42dbbe3134a079be54a9256e5c5a3b0"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.300470 10866 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:15:28.301158 10881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.301326 10866 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:15:28.301432 10866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "368b74b9f631d697d5b6c3fc084772d6"
server_key_iv: "c42dbbe3134a079be54a9256e5c5a3b0"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.301505 10866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:28.314250 10866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:28.316130 10866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:28.316298 10866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:28.316555 10866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:28.316874 10866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:28.316936 10866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.316998 10866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:28.317030 10866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.323815 10866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42511
I20260926 13:15:28.323859 10994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42511 every 8 connection(s)
I20260926 13:15:28.324186 10866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:15:28.326471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 10866
I20260926 13:15:28.326578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:15:28.326768 392 external_mini_cluster.cc:1428] Setting key 1ca15e93dc1bfcbdff9ce9d6226d58fc
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:15:28.334652 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.324949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50325 (local address 127.0.98.62:43385)
0926 13:15:28.325026 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.325028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.325240 (+ 212us) server_negotiation.cc:407] Connection header received
0926 13:15:28.325754 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.325755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.325765 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.325826 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:28.326995 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.327621 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.328650 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.328859 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.328947 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.331483 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:28.331484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:28.331485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:28.331493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:28.333159 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.333627 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.333628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.333629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.333694 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.333946 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.333947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.333948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.334081 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:28.334161 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.334468 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.334555 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 13:15:28.334883 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.324921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43385 (local address 127.0.98.2:50325)
0926 13:15:28.325105 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.325148 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.325637 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.325915 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.325919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.326628 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:28.326888 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.326894 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.327712 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.327713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.328539 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.328542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.328955 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.328956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.329102 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.329671 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:28.329680 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:28.331373 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:28.333254 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:28.333261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:28.333263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:28.333500 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:28.333765 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:28.333766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:28.333766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:28.333874 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:28.334152 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:28.334158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:28.334331 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.334447 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.334621 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
I20260926 13:15:28.335398 10995 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43385
I20260926 13:15:28.335513 10995 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:28.335721 10995 heartbeater.cc:511] Master 127.0.98.62:43385 requested a full tablet report, sending...
I20260926 13:15:28.336112 10670 ts_manager.cc:194] Registered new tserver with Master: 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc (127.0.98.2:42511)
I20260926 13:15:28.336457 10670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:15:28.352811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:15:28.435360 11000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:15:28.435565 11000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:15:28.435587 11000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:15:28.436652 11000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:15:28.436685 11000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:15:28.436712 11000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:15:28.438139 11000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:15:28.438355 11000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:15:28.438519 11000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:15:28.443444 11005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:28.443575 11000 server_base.cc:1034] running on GCE node
W20260926 13:15:28.443486 11006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:15:28.443456 11008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:15:28.443890 11000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:15:28.444140 11000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:15:28.445298 11000 hybrid_clock.cc:648] HybridClock initialized: now 1790428528445297 us; error 33 us; skew 500 ppm
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[10625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:15:28.447752 11000 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:15:28.448212 11000 webserver.cc:460] Webserver started at http://127.0.98.3:37385/ using document root <none> and password file <none>
I20260926 13:15:28.448449 11000 fs_manager.cc:359] Metadata directory not provided
I20260926 13:15:28.448514 11000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:15:28.448643 11000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:15:28.449658 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "831aa00d24d34de387bf70c7c5be36c4"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "0876416d5df5f9369f5d2affaf686c71"
server_key_iv: "23a1bed8247e50ef998e3e50c7ed8e8a"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.450024 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "831aa00d24d34de387bf70c7c5be36c4"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "0876416d5df5f9369f5d2affaf686c71"
server_key_iv: "23a1bed8247e50ef998e3e50c7ed8e8a"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.451318 11000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:15:28.451993 11015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.452164 11000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:15:28.452271 11000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "831aa00d24d34de387bf70c7c5be36c4"
format_stamp: "Formatted at 2026-09-26 13:15:28 on dist-test-slave-f864"
server_key: "0876416d5df5f9369f5d2affaf686c71"
server_key_iv: "23a1bed8247e50ef998e3e50c7ed8e8a"
server_key_version: "encryptionkey@0"
I20260926 13:15:28.452353 11000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:15:28.463542 11000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:15:28.465265 11000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:15:28.465425 11000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:15:28.465639 11000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:15:28.465938 11000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:15:28.465991 11000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.466030 11000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:15:28.466059 11000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:15:28.471944 11000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33143
I20260926 13:15:28.471985 11128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33143 every 8 connection(s)
I20260926 13:15:28.472311 11000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:15:28.477550 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 11000
I20260926 13:15:28.477655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:15:28.477844 392 external_mini_cluster.cc:1428] Setting key 225c6b4777dfd31cb57700d58542465b
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[10625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:15:28.481618 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.473032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39427 (local address 127.0.98.62:43385)
0926 13:15:28.473111 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.473114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.473268 (+ 154us) server_negotiation.cc:407] Connection header received
0926 13:15:28.473712 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.473713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.473724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.473776 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:28.474838 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.475437 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.476599 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.476791 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.476849 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.479369 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:28.479370 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:28.479370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:28.479376 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:28.480515 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.480882 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.480883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.480883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.480933 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.481103 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.481103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.481104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.481198 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:28.481247 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.481473 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.481532 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:15:28.481752 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.472981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43385 (local address 127.0.98.3:39427)
0926 13:15:28.473140 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.473174 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.473634 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.473826 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.473829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.474466 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:15:28.474755 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.474761 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.475512 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.475513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.476488 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.476491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.476874 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.476875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.477003 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.477496 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:15:28.477507 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:15:28.479290 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:15:28.480597 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:28.480603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:28.480604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:28.480819 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:28.480973 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:15:28.480973 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:15:28.480973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:15:28.481055 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:15:28.481253 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:15:28.481258 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:15:28.481367 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.481448 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.481561 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":101,"threads_started":1}
I20260926 13:15:28.482293 11129 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43385
I20260926 13:15:28.482394 11129 heartbeater.cc:464] Registering TS with master...
I20260926 13:15:28.482578 11129 heartbeater.cc:511] Master 127.0.98.62:43385 requested a full tablet report, sending...
I20260926 13:15:28.482950 10670 ts_manager.cc:194] Registered new tserver with Master: 831aa00d24d34de387bf70c7c5be36c4 (127.0.98.3:33143)
I20260926 13:15:28.483327 10670 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39427
I20260926 13:15:28.489393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:15:28.496148 10710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.490661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.62:43385)
0926 13:15:28.490733 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.490735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.490786 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:15:28.490838 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.490839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.490849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.490913 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:15:28.491404 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.491899 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.492610 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.492793 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.492855 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.493984 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:15:28.493985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:15:28.493985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:15:28.493991 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:15:28.495292 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.495576 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.495577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.495577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.495625 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:15:28.495768 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:15:28.495768 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:15:28.495769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:15:28.495881 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:15:28.495960 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.496036 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.496090 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 13:15:28.497711 10670 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:15:28.499441 10670 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:15:28.505683 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.501587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.0.98.2:42511)
0926 13:15:28.502018 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.502022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.502031 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:15:28.502101 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.502103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.502119 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.502414 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:28.502483 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.503127 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.504549 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.505218 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.505389 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.505453 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.505557 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.505600 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":94,"threads_started":1}
I20260926 13:15:28.505673 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.501480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42511 (local address 127.0.0.1:57510)
0926 13:15:28.501663 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.501701 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.501763 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.502262 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.502265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.502267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:28.502335 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.502337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.503234 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.503235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.504453 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.504455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.505377 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.505379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.505453 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.505466 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.505542 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.505600 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1,"thread_start_us":97,"threads_started":1}
I20260926 13:15:28.505999 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.501816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59424 (local address 127.0.98.3:33143)
0926 13:15:28.502031 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.502036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.502046 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:15:28.502084 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.502086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.502102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.502204 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:28.502424 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.503532 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.504821 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.505496 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.505556 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.505607 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.505869 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.505909 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":93,"threads_started":1}
I20260926 13:15:28.506122 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.501816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.98.1:36175)
0926 13:15:28.502031 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.502036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.502057 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:15:28.502092 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.502094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.502107 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.502247 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:28.502424 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.503532 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.504739 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.505487 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.505557 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.505608 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.505991 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.506039 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260926 13:15:28.505918 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.501704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36175 (local address 127.0.0.1:46072)
0926 13:15:28.501862 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.501872 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.501910 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.502316 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.502318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.502319 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:28.502351 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.502353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.503642 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.503643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.504658 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.504659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.505646 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.505646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.505734 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.505738 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.505784 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.505827 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":29,"thread_start_us":50,"threads_started":1}
I20260926 13:15:28.506804 10929 tablet_service.cc:1467] Processing CreateTablet for tablet 657eca343cca49d080031a48d26df15a (DEFAULT_TABLE table=test-table [id=82415ea8287141dda9288f122a571222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:28.506804 10795 tablet_service.cc:1467] Processing CreateTablet for tablet 657eca343cca49d080031a48d26df15a (DEFAULT_TABLE table=test-table [id=82415ea8287141dda9288f122a571222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:28.507165 10795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657eca343cca49d080031a48d26df15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:28.507165 10929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657eca343cca49d080031a48d26df15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:28.505975 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.501480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33143 (local address 127.0.0.1:59424)
0926 13:15:28.501790 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.501802 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.501856 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.502283 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.502285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.502287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:28.502345 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.502346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.503676 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.503677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.504750 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.504753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.505645 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.505646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.505730 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.505734 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.505784 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.505827 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":6,"thread_start_us":63,"threads_started":1}
I20260926 13:15:28.510514 11144 tablet_bootstrap.cc:492] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Bootstrap starting.
I20260926 13:15:28.510877 11143 tablet_bootstrap.cc:492] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: Bootstrap starting.
I20260926 13:15:28.510855 11063 tablet_service.cc:1467] Processing CreateTablet for tablet 657eca343cca49d080031a48d26df15a (DEFAULT_TABLE table=test-table [id=82415ea8287141dda9288f122a571222]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:15:28.511159 11063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657eca343cca49d080031a48d26df15a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:15:28.511194 11144 tablet_bootstrap.cc:654] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:28.511511 11143 tablet_bootstrap.cc:654] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:28.511786 11144 log.cc:824] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:28.512064 11143 log.cc:824] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:28.512626 11144 tablet_bootstrap.cc:492] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: No bootstrap required, opened a new log
I20260926 13:15:28.512676 11144 ts_tablet_manager.cc:1397] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:15:28.513957 11147 tablet_bootstrap.cc:492] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Bootstrap starting.
I20260926 13:15:28.514274 11144 raft_consensus.cc:348] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.514461 11144 raft_consensus.cc:374] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:28.514472 11143 tablet_bootstrap.cc:492] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: No bootstrap required, opened a new log
I20260926 13:15:28.514498 11144 raft_consensus.cc:729] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc, State: Initialized, Role: FOLLOWER
I20260926 13:15:28.514530 11143 ts_tablet_manager.cc:1397] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 13:15:28.514614 11147 tablet_bootstrap.cc:654] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Neither blocks nor log segments found. Creating new log.
I20260926 13:15:28.514619 11144 consensus_queue.cc:260] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.514922 11144 ts_tablet_manager.cc:1428] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:15:28.514954 10995 heartbeater.cc:503] Master 127.0.98.62:43385 was elected leader, sending a full tablet report...
I20260926 13:15:28.515228 11147 log.cc:824] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Log is configured to *not* fsync() on all Append() calls
I20260926 13:15:28.515949 11147 tablet_bootstrap.cc:492] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: No bootstrap required, opened a new log
I20260926 13:15:28.516024 11147 ts_tablet_manager.cc:1397] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:15:28.516637 11143 raft_consensus.cc:348] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.516813 11143 raft_consensus.cc:374] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:28.516856 11143 raft_consensus.cc:729] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf840a09ed044f78d5da33de3cb63a1, State: Initialized, Role: FOLLOWER
I20260926 13:15:28.516983 11143 consensus_queue.cc:260] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.517266 11143 ts_tablet_manager.cc:1428] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:15:28.517338 10861 heartbeater.cc:503] Master 127.0.98.62:43385 was elected leader, sending a full tablet report...
I20260926 13:15:28.518024 11147 raft_consensus.cc:348] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.518229 11147 raft_consensus.cc:374] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:15:28.518272 11147 raft_consensus.cc:729] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 831aa00d24d34de387bf70c7c5be36c4, State: Initialized, Role: FOLLOWER
I20260926 13:15:28.518385 11147 consensus_queue.cc:260] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.518632 11147 ts_tablet_manager.cc:1428] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:15:28.518682 11129 heartbeater.cc:503] Master 127.0.98.62:43385 was elected leader, sending a full tablet report...
I20260926 13:15:28.573097 11151 raft_consensus.cc:482] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:15:28.573244 11151 raft_consensus.cc:504] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.573583 11151 leader_election.cc:288] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bf840a09ed044f78d5da33de3cb63a1 (127.0.98.1:36175), 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc (127.0.98.2:42511)
W20260926 13:15:28.574898 10996 tablet.cc:2369] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:15:28.576927 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.573882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37549 (local address 127.0.98.1:36175)
0926 13:15:28.573928 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.573931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.574143 (+ 212us) server_negotiation.cc:407] Connection header received
0926 13:15:28.574195 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.574197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.574212 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.574296 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:28.574445 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.575040 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.576153 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.576584 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.576660 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.576670 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.576821 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.576861 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:15:28.576942 11152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.573790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36175 (local address 127.0.98.3:37549)
0926 13:15:28.574052 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.574060 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.574119 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.574308 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.574309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.574310 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:28.574367 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.574369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.575153 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.575154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.576059 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.576061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.576671 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.576672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.576740 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.576744 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.576815 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.576854 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":56,"thread_start_us":117,"threads_started":1}
I20260926 13:15:28.577165 11131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.573790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42511 (local address 127.0.98.3:35337)
0926 13:15:28.573867 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.573882 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 13:15:28.573931 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:15:28.574259 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:15:28.574260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:15:28.574262 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:15:28.574367 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.574369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.575115 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.575116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.576021 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:15:28.576024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.576714 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.576715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:15:28.576802 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.576805 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:15:28.576867 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:15:28.576907 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 13:15:28.577132 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.573992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35337 (local address 127.0.98.2:42511)
0926 13:15:28.574070 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.574072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.574084 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:15:28.574114 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.574115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.574128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.574436 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:15:28.574473 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.575043 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.576103 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.576634 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.576715 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.576725 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.576884 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.576919 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:15:28.577307 10815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657eca343cca49d080031a48d26df15a" candidate_uuid: "831aa00d24d34de387bf70c7c5be36c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf840a09ed044f78d5da33de3cb63a1" is_pre_election: true
I20260926 13:15:28.577661 10815 raft_consensus.cc:2454] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831aa00d24d34de387bf70c7c5be36c4 in term 0.
I20260926 13:15:28.577647 10949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657eca343cca49d080031a48d26df15a" candidate_uuid: "831aa00d24d34de387bf70c7c5be36c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" is_pre_election: true
I20260926 13:15:28.577785 10949 raft_consensus.cc:2454] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 831aa00d24d34de387bf70c7c5be36c4 in term 0.
I20260926 13:15:28.577903 11018 leader_election.cc:302] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bf840a09ed044f78d5da33de3cb63a1, 831aa00d24d34de387bf70c7c5be36c4; no voters:
I20260926 13:15:28.578028 11151 raft_consensus.cc:2781] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:15:28.578081 11151 raft_consensus.cc:482] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:15:28.578111 11151 raft_consensus.cc:3037] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:28.578842 11151 raft_consensus.cc:504] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.578992 11151 leader_election.cc:288] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [CANDIDATE]: Term 1 election: Requested vote from peers 1bf840a09ed044f78d5da33de3cb63a1 (127.0.98.1:36175), 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc (127.0.98.2:42511)
I20260926 13:15:28.579135 10815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657eca343cca49d080031a48d26df15a" candidate_uuid: "831aa00d24d34de387bf70c7c5be36c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bf840a09ed044f78d5da33de3cb63a1"
I20260926 13:15:28.579154 10949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657eca343cca49d080031a48d26df15a" candidate_uuid: "831aa00d24d34de387bf70c7c5be36c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc"
I20260926 13:15:28.579224 10949 raft_consensus.cc:3037] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:28.579219 10815 raft_consensus.cc:3037] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:15:28.580096 10815 raft_consensus.cc:2454] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831aa00d24d34de387bf70c7c5be36c4 in term 1.
I20260926 13:15:28.580099 10949 raft_consensus.cc:2454] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 831aa00d24d34de387bf70c7c5be36c4 in term 1.
I20260926 13:15:28.580293 11017 leader_election.cc:302] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc, 831aa00d24d34de387bf70c7c5be36c4; no voters:
I20260926 13:15:28.580405 11151 raft_consensus.cc:2781] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:15:28.580555 11151 raft_consensus.cc:686] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 1 LEADER]: Becoming Leader. State: Replica: 831aa00d24d34de387bf70c7c5be36c4, State: Running, Role: LEADER
I20260926 13:15:28.580659 11151 consensus_queue.cc:237] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } }
I20260926 13:15:28.581537 10670 catalog_manager.cc:5495] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 831aa00d24d34de387bf70c7c5be36c4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "831aa00d24d34de387bf70c7c5be36c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "831aa00d24d34de387bf70c7c5be36c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20260926 13:15:28.597308 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.594845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59428 (local address 127.0.98.3:33143)
0926 13:15:28.594908 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:15:28.594911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:15:28.594920 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:15:28.594981 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:15:28.594982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:15:28.595012 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:15:28.595088 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:15:28.595239 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.595699 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.596486 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:15:28.596681 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:15:28.596762 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:15:28.596936 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:15:28.597043 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:15:28.597141 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:15:28.597159 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 13:15:28.597183 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:15:28.599318 10815 raft_consensus.cc:1261] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1 [term 1 FOLLOWER]: Refusing update from remote peer 831aa00d24d34de387bf70c7c5be36c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:28.599318 10949 raft_consensus.cc:1261] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 1 FOLLOWER]: Refusing update from remote peer 831aa00d24d34de387bf70c7c5be36c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:15:28.599751 11151 consensus_queue.cc:1035] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bf840a09ed044f78d5da33de3cb63a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:28.599920 11153 consensus_queue.cc:1035] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c7a7ff4d5ad40ae9ad2ce7559f4a8cc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:15:28.603324 11157 mvcc.cc:205] Tried to move back new op lower bound from 7333595253139951616 to 7333595253067231232. Current Snapshot: MvccSnapshot[applied={T|T < 7333595253139951616}]
I20260926 13:15:28.605664 10669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:28.605763 10669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:15:28.606627 10669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9820bb1bc0b847e09d7720cad1d66685: Sending DeleteTablet for 3 replicas of tablet 657eca343cca49d080031a48d26df15a
I20260926 13:15:28.606986 10795 tablet_service.cc:1514] Processing DeleteTablet for tablet 657eca343cca49d080031a48d26df15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46072
I20260926 13:15:28.606987 10929 tablet_service.cc:1514] Processing DeleteTablet for tablet 657eca343cca49d080031a48d26df15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57510
I20260926 13:15:28.607295 11165 tablet_replica.cc:330] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: stopping tablet replica
I20260926 13:15:28.607270 11063 tablet_service.cc:1514] Processing DeleteTablet for tablet 657eca343cca49d080031a48d26df15a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:15:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59424
I20260926 13:15:28.607401 11165 raft_consensus.cc:2229] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:15:28.607514 11165 raft_consensus.cc:2258] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:28.607525 11167 tablet_replica.cc:330] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: stopping tablet replica
I20260926 13:15:28.607671 11166 tablet_replica.cc:330] T 657eca343cca49d080031a48d26df15a P 1bf840a09ed044f78d5da33de3cb63a1: stopping tablet replica
I20260926 13:15:28.607710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10732
I20260926 13:15:28.607836 11167 raft_consensus.cc:2229] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:15:28.607985 11167 raft_consensus.cc:2258] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:15:28.608676 11167 ts_tablet_manager.cc:1905] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:28.610574 11165 ts_tablet_manager.cc:1905] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:15:28.612233 11167 ts_tablet_manager.cc:1918] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:28.612424 11165 ts_tablet_manager.cc:1918] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:15:28.612532 11165 log.cc:1196] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/657eca343cca49d080031a48d26df15a
I20260926 13:15:28.612632 11167 log.cc:1196] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/657eca343cca49d080031a48d26df15a
I20260926 13:15:28.612990 11165 ts_tablet_manager.cc:1939] T 657eca343cca49d080031a48d26df15a P 1c7a7ff4d5ad40ae9ad2ce7559f4a8cc: Deleting consensus metadata
I20260926 13:15:28.613327 11167 ts_tablet_manager.cc:1939] T 657eca343cca49d080031a48d26df15a P 831aa00d24d34de387bf70c7c5be36c4: Deleting consensus metadata
W20260926 13:15:28.613607 10656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36175 (error 108)
I20260926 13:15:28.613863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10866
I20260926 13:15:28.614497 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.613971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36175 (local address 127.0.0.1:46080)
0926 13:15:28.614411 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.614462 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260926 13:15:28.615278 10656 catalog_manager.cc:4584] TS 1bf840a09ed044f78d5da33de3cb63a1 (127.0.98.1:36175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 657eca343cca49d080031a48d26df15a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36175: connect: Connection refused (error 111)
I20260926 13:15:28.615605 10656 catalog_manager.cc:4841] TS 831aa00d24d34de387bf70c7c5be36c4 (127.0.98.3:33143): tablet 657eca343cca49d080031a48d26df15a (table test-table [id=82415ea8287141dda9288f122a571222]) successfully deleted
I20260926 13:15:28.619359 11140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:15:28.619222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42511 (local address 127.0.0.1:57516)
0926 13:15:28.619286 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 13:15:28.619320 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:15:28.619431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 11000
I20260926 13:15:28.623986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 10638
2026-09-26T13:15:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1951 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[11171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[11171](info): set up 2 sockets
Sep 26 13:15:28 dist-test-slave-f864 krb5kdc[11171](info): commencing operation
krb5kdc: starting...
W20260926 13:15:29.670399 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:15:29 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:15:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 13:15:29.835250 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLNUYBk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/postgres -l logfile start
2026-09-26 13:15:32.398 UTC [11201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:15:32.398 UTC [11201] LOG: listening on IPv4 address "127.0.98.20", port 32795
2026-09-26 13:15:32.402 UTC [11201] LOG: listening on Unix socket "/tmp/.s.PGSQL.32795"
2026-09-26 13:15:32.424 UTC [11204] LOG: database system was shut down at 2026-09-26 13:15:30 UTC
2026-09-26 13:15:32.428 UTC [11201] LOG: database system is ready to accept connections
I20260926 13:15:33.383389 392 mini_postgres.cc:96] Postgres bound to 32795
2026-09-26 13:15:33.391 UTC [11212] FATAL: database "slave" does not exist
127.0.98.20:32795 - accepting connections
I20260926 13:15:33.391863 392 mini_ranger.cc:161] Starting Ranger...
I20260926 13:15:33.410843 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:15:33.902514 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:15:33.902583 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-admin
2026-09-26 13:15:34,185 [I] DB FLAVOR :POSTGRES
2026-09-26 13:15:34,185 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:15:34,185 [I] Checking connection..
2026-09-26 13:15:34,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:15:34,429 [I] Checking connection passed.
2026-09-26 13:15:34,429 [I] --------- Verifying version history table ---------
2026-09-26 13:15:34,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:15:34,686 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:15:34,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:15:34,952 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:15:34,953 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:15:34,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:15:35.183 UTC [11276] WARNING: there is no transaction in progress
2026-09-26 13:15:35,189 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:15:35,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:15:35,448 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:15:35,449 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:15:35,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:15:35,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:15:35,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f864.c.gcp-upstream.internal','N') ;"
2026-09-26 13:15:36,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:15:36,389 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:15:36,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:15:36,621 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:15:36,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:15:36,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:15:37,095 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:15:37,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:15:38.051 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.067 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.082 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.098 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.116 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.137 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.192 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.199 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.214 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.225 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.235 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.249 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.259 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.265 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.272 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38.279 UTC [11384] WARNING: there is no transaction in progress
2026-09-26 13:15:38,947 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:15:38,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:15:39,197 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:15:39,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:15:39,435 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:15:39,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:15:39,684 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:15:39,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:15:39,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:15:40,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f864.c.gcp-upstream.internal';"
2026-09-26 13:15:40,409 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:15:40,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:15:40,654 [I] DB_PATCHES have already been applied
I20260926 13:15:40.660187 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLNUYBk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:15:40.660310 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 13:15:40.663447 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:35391
Sep 26, 2026 1:15:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:15:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:15:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:15:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:15:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:15:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:15:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:15:41.768623 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.377s user 0.000s sys 0.010s
Sep 26, 2026 1:16:02 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:16:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:16:02 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:16:02 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:16:03 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:16:03.781793 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:16:03 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:16:03.866510 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:16:03.875694 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:16:04.738250 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:16:04.738337 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms
I20260926 13:16:04.738359 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:32795
2026-09-26 13:16:04,918 [I] DB FLAVOR :POSTGRES
2026-09-26 13:16:04,918 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:16:04,918 [I] Checking connection
2026-09-26 13:16:04,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:16:05,180 [I] connection success
2026-09-26 13:16:05,181 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:16:05,181 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:16:05,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:16:05,416 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:16:05,417 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:16:05,417 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:16:05,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:32795/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:16:05,677 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:16:06.123494 392 mini_ranger_kms.cc:310] Created kms service
I20260926 13:16:06.289362 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:16:06.331429 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:16:06 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428563, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:06.740777 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:16:06.740891 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:16:06.740916 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 13:16:06.744604 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11564
I20260926 13:16:06.744760 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:48983
Sep 26, 2026 1:16:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:16:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:16:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:16:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:16:08 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:16:08 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 13:16:08 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428568, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:09.051482 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.185s user 0.000s sys 0.162s
I20260926 13:16:09.051605 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:16:09.051708 392 mini_ranger_kms.cc:401] 127.0.98.20:48983/kms/v1/keys
Sep 26 13:16:09 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428568, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:16:09 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428568, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:16:12 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:16:12.125295 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39073
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39073
--ranger_config_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48983/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:16:12.231326 11615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:16:12.231532 11615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:16:12.231554 11615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:16:12.232678 11615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:16:12.232720 11615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:16:12.232735 11615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:16:12.232748 11615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:16:12.232759 11615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:16:12.234341 11615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:48983/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39073
--ranger_config_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:16:12.234587 11615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:16:12.234803 11615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:16:12.240557 11620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:12.240549 11621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:12.240548 11623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:12.240772 11615 server_base.cc:1034] running on GCE node
I20260926 13:16:12.241114 11615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:16:12.241410 11615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:16:12.242563 11615 hybrid_clock.cc:648] HybridClock initialized: now 1790428572242535 us; error 43 us; skew 500 ppm
Sep 26 13:16:12 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:12.250984 11615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:16:12.251502 11615 webserver.cc:460] Webserver started at http://127.0.98.62:44517/ using document root <none> and password file <none>
I20260926 13:16:12.251735 11615 fs_manager.cc:359] Metadata directory not provided
I20260926 13:16:12.251778 11615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:16:12.251919 11615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:16:12 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:12.310149 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "474cd5ac17ab485694ab77cdcc5757d7"
format_stamp: "Formatted at 2026-09-26 13:16:12 on dist-test-slave-f864"
server_key: "bb3564701cac969d85994f186e333ed9"
server_key_iv: "be6934291654b387ba7fdcdf3993a499"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:12.310702 11615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "474cd5ac17ab485694ab77cdcc5757d7"
format_stamp: "Formatted at 2026-09-26 13:16:12 on dist-test-slave-f864"
server_key: "bb3564701cac969d85994f186e333ed9"
server_key_iv: "be6934291654b387ba7fdcdf3993a499"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:12.312491 11615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 13:16:12.327723 11630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:12.327903 11615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:16:12.328001 11615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "474cd5ac17ab485694ab77cdcc5757d7"
format_stamp: "Formatted at 2026-09-26 13:16:12 on dist-test-slave-f864"
server_key: "bb3564701cac969d85994f186e333ed9"
server_key_iv: "be6934291654b387ba7fdcdf3993a499"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:12.328105 11615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:16:12.348245 11615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:16:12.350446 11615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:16:12.350565 11615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:16:12.355078 11615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39073
I20260926 13:16:12.355182 11682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39073 every 8 connection(s)
I20260926 13:16:12.355500 11615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:16:12.356189 11683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:16:12.358451 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7: Bootstrap starting.
I20260926 13:16:12.358966 11683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7: Neither blocks nor log segments found. Creating new log.
I20260926 13:16:12.359575 11683 log.cc:824] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7: Log is configured to *not* fsync() on all Append() calls
I20260926 13:16:12.360244 11683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7: No bootstrap required, opened a new log
I20260926 13:16:12.362077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 11615
I20260926 13:16:12.362202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:16:12.362110 11683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474cd5ac17ab485694ab77cdcc5757d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260926 13:16:12.362353 11683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:16:12.362397 392 external_mini_cluster.cc:1428] Setting key 911f4e5a3686bcb7afb36532441914f3
I20260926 13:16:12.362423 11683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 474cd5ac17ab485694ab77cdcc5757d7, State: Initialized, Role: FOLLOWER
I20260926 13:16:12.362605 11683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474cd5ac17ab485694ab77cdcc5757d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260926 13:16:12.362695 11683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:16:12.362740 11683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:16:12.362795 11683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:16:12.363502 11683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474cd5ac17ab485694ab77cdcc5757d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260926 13:16:12.363605 11683 leader_election.cc:302] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 474cd5ac17ab485694ab77cdcc5757d7; no voters:
I20260926 13:16:12.363761 11683 leader_election.cc:288] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:16:12.363783 11688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:16:12.363894 11688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [term 1 LEADER]: Becoming Leader. State: Replica: 474cd5ac17ab485694ab77cdcc5757d7, State: Running, Role: LEADER
I20260926 13:16:12.364022 11683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:16:12.364039 11688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474cd5ac17ab485694ab77cdcc5757d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } }
I20260926 13:16:12.364447 11689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 474cd5ac17ab485694ab77cdcc5757d7. Latest consensus state: current_term: 1 leader_uuid: "474cd5ac17ab485694ab77cdcc5757d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474cd5ac17ab485694ab77cdcc5757d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } } }
I20260926 13:16:12.364527 11689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:16:12.364732 11696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:16:12.365216 11696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:16:12.365266 11690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "474cd5ac17ab485694ab77cdcc5757d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "474cd5ac17ab485694ab77cdcc5757d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39073 } } }
I20260926 13:16:12.365360 11690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7 [sys.catalog]: This master's current role is: LEADER
I20260926 13:16:12.365830 11683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 26 13:16:12 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:16:12.374284 11696 catalog_manager.cc:1347] Generated new cluster ID: f7edcc3ab4f1419189be2626e9969726
I20260926 13:16:12.374349 11696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:16:12.378252 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:12.362933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.98.62:39073)
0926 13:16:12.363162 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:12.363171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:12.363205 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:16:12.363320 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:12.363328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:12.363391 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:12.363545 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:12.364355 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:12.365807 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:12.366846 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:12.367274 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:12.367417 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:12.371515 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:12.371526 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:12.371526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:12.371544 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:12.373736 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:12.374218 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:12.374223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:12.374223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:12.374352 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:12.377154 (+ 2802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:12.377155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:12.377155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:12.377518 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:12.377674 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:12.377825 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 13:16:12.377986 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260926 13:16:12.392078 11696 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:16:12.392925 11696 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:16:12.400887 11696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7: Generated new TSK 0
I20260926 13:16:12.401114 11696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:16:13.069624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:16:13.177376 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:16:13.177594 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:16:13.177632 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:16:13.178781 11728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:16:13.178845 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:16:13.178889 11728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:16:13.180377 11728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:48983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:16:13.180640 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:16:13.180894 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:16:13.185676 11733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:13.185770 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:13.186096 11734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:13.186136 11728 server_base.cc:1034] running on GCE node
I20260926 13:16:13.186406 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:16:13.186676 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:16:13.187829 11728 hybrid_clock.cc:648] HybridClock initialized: now 1790428573187813 us; error 37 us; skew 500 ppm
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:13.190402 11728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:16:13.190917 11728 webserver.cc:460] Webserver started at http://127.0.98.1:41787/ using document root <none> and password file <none>
I20260926 13:16:13.191169 11728 fs_manager.cc:359] Metadata directory not provided
I20260926 13:16:13.191214 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:16:13.191341 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:13.212308 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "93c1061774d54a359f4761c09b66c71f"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "646f5ac6ebe3ca01ab57e3487a86e625"
server_key_iv: "122a3e5c1d46a6fbe1627a085783ed8c"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.212764 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93c1061774d54a359f4761c09b66c71f"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "646f5ac6ebe3ca01ab57e3487a86e625"
server_key_iv: "122a3e5c1d46a6fbe1627a085783ed8c"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.214460 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 13:16:13.236418 11745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.236631 11728 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260926 13:16:13.236727 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "93c1061774d54a359f4761c09b66c71f"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "646f5ac6ebe3ca01ab57e3487a86e625"
server_key_iv: "122a3e5c1d46a6fbe1627a085783ed8c"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.236833 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:16:13.256793 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:16:13.258934 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:16:13.259109 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:16:13.259361 11728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:16:13.259730 11728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:16:13.259783 11728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.259851 11728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:16:13.259894 11728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.267613 11728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45849
I20260926 13:16:13.267704 11858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45849 every 8 connection(s)
I20260926 13:16:13.268033 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:16:13.277051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 11728
I20260926 13:16:13.277146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:16:13.277354 392 external_mini_cluster.cc:1428] Setting key 4e4570ecc1c9e02b817dc96250accc0f
I20260926 13:16:13.278111 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.268838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39073 (local address 127.0.98.1:38701)
0926 13:16:13.269022 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.269070 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.269594 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.269888 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.269892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.270084 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:16:13.270348 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.270354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.271050 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.271051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.272080 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.272082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.272491 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.272492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.272628 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.273183 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:16:13.273191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:16:13.274601 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:16:13.276433 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:13.276439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:13.276442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:13.276718 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:13.276970 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:13.276971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:13.276971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:13.277117 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:13.277431 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:16:13.277437 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:16:13.277620 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.277741 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.277883 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 13:16:13.277945 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.268867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38701 (local address 127.0.98.62:39073)
0926 13:16:13.269020 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.269022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.269175 (+ 153us) server_negotiation.cc:407] Connection header received
0926 13:16:13.269730 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.269732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.269746 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.269818 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:13.270450 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.270959 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.272175 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.272397 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.272482 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.274713 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:13.274715 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:13.274715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:13.274722 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:13.276329 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.276807 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.276808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.276809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.276894 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.277194 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.277195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.277196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.277342 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:13.277404 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.277766 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.277853 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260926 13:16:13.280117 11859 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39073
I20260926 13:16:13.280239 11859 heartbeater.cc:464] Registering TS with master...
I20260926 13:16:13.280423 11859 heartbeater.cc:511] Master 127.0.98.62:39073 requested a full tablet report, sending...
I20260926 13:16:13.280898 11647 ts_manager.cc:194] Registered new tserver with Master: 93c1061774d54a359f4761c09b66c71f (127.0.98.1:45849)
I20260926 13:16:13.281503 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38701
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:16:13.313699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 13:16:13.425235 11866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:16:13.425592 11866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:16:13.425673 11866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:16:13.427423 11866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:16:13.427531 11866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:16:13.427626 11866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:16:13.429664 11866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:48983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:16:13.430080 11866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:16:13.430388 11866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:16:13.436520 11873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:13.436586 11875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:13.436740 11866 server_base.cc:1034] running on GCE node
W20260926 13:16:13.436911 11872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:13.437148 11866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:16:13.437422 11866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:16:13.438581 11866 hybrid_clock.cc:648] HybridClock initialized: now 1790428573438567 us; error 35 us; skew 500 ppm
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:13.441565 11866 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:16:13.441984 11866 webserver.cc:460] Webserver started at http://127.0.98.2:34855/ using document root <none> and password file <none>
I20260926 13:16:13.442276 11866 fs_manager.cc:359] Metadata directory not provided
I20260926 13:16:13.442346 11866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:16:13.442492 11866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:13.455219 11866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "810ab0db3353ec25b1268dcf3d7748d4"
server_key_iv: "f8c23224462903bef1a9da3c40f6d6e8"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.455701 11866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "810ab0db3353ec25b1268dcf3d7748d4"
server_key_iv: "f8c23224462903bef1a9da3c40f6d6e8"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.461895 11866 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260926 13:16:13.474764 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.475016 11866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:16:13.475118 11866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "810ab0db3353ec25b1268dcf3d7748d4"
server_key_iv: "f8c23224462903bef1a9da3c40f6d6e8"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.475210 11866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:16:13.497385 11866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:16:13.500048 11866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:16:13.500203 11866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:16:13.500423 11866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:16:13.500808 11866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:16:13.500854 11866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.500885 11866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:16:13.500910 11866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.507625 11866 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35189
I20260926 13:16:13.508057 11866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:16:13.511207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 11866
I20260926 13:16:13.511320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:16:13.511515 392 external_mini_cluster.cc:1428] Setting key ab209af11979c60f9b0ca7e5175d62fe
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:16:13.509424 11996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35189 every 8 connection(s)
I20260926 13:16:13.521317 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.508841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39073 (local address 127.0.98.2:46251)
0926 13:16:13.509736 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.509785 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.510426 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.510797 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.510801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.511047 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:16:13.512010 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.512017 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.513069 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.513070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.514389 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.514392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.514960 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.514961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.515095 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.515712 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:16:13.515724 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:16:13.517468 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:16:13.519608 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:13.519614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:13.519617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:13.519913 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:13.520145 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:13.520145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:13.520146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:13.520282 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:13.520618 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:16:13.520625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:16:13.520809 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.520921 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.521078 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":88,"threads_started":1}
I20260926 13:16:13.521893 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.508866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46251 (local address 127.0.98.62:39073)
0926 13:16:13.508917 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.508919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.509905 (+ 986us) server_negotiation.cc:407] Connection header received
0926 13:16:13.510553 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.510555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.510568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.510599 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:13.512121 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.512886 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.514493 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.514823 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.514867 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.517581 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:13.517582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:13.517583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:13.517591 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:13.519508 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.520005 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.520006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.520007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.520069 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.520358 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.520359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.520359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.520518 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:13.520587 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.521709 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.521806 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:16:13.522574 11997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39073
I20260926 13:16:13.522718 11997 heartbeater.cc:464] Registering TS with master...
I20260926 13:16:13.522964 11997 heartbeater.cc:511] Master 127.0.98.62:39073 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:16:13.523432 11647 ts_manager.cc:194] Registered new tserver with Master: 2ebe42baeb0a41ba9a1f75d74c44ccb8 (127.0.98.2:35189)
I20260926 13:16:13.523918 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:16:13.546077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:48983/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:16:13.678946 12002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:16:13.679273 12002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:16:13.679316 12002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:16:13.680532 12002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:16:13.680608 12002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:16:13.680656 12002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:16:13.682348 12002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:48983/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:16:13.682613 12002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:16:13.682849 12002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:16:13.688710 12007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:13.688704 12008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:13.688731 12010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:13.688927 12002 server_base.cc:1034] running on GCE node
I20260926 13:16:13.689163 12002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:16:13.689425 12002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:16:13.690582 12002 hybrid_clock.cc:648] HybridClock initialized: now 1790428573690556 us; error 34 us; skew 500 ppm
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:13.693941 12002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:16:13.694451 12002 webserver.cc:460] Webserver started at http://127.0.98.3:35987/ using document root <none> and password file <none>
I20260926 13:16:13.694679 12002 fs_manager.cc:359] Metadata directory not provided
I20260926 13:16:13.694748 12002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:16:13.694895 12002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:13.719118 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "29c83a6bee2747aebf11f550c22ea9ac"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "0eab9a21c707f6d98f804ef6847e8965"
server_key_iv: "eb8e2a45899a9a40f6aad459e846692b"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.719592 12002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "29c83a6bee2747aebf11f550c22ea9ac"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "0eab9a21c707f6d98f804ef6847e8965"
server_key_iv: "eb8e2a45899a9a40f6aad459e846692b"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.721354 12002 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 13:16:13.743384 12017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.743610 12002 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.001s
I20260926 13:16:13.743709 12002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "29c83a6bee2747aebf11f550c22ea9ac"
format_stamp: "Formatted at 2026-09-26 13:16:13 on dist-test-slave-f864"
server_key: "0eab9a21c707f6d98f804ef6847e8965"
server_key_iv: "eb8e2a45899a9a40f6aad459e846692b"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:13.743822 12002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:16:13.771226 12002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:16:13.773948 12002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:16:13.774147 12002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:16:13.774482 12002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:16:13.774928 12002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:16:13.774998 12002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.775041 12002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:16:13.775072 12002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:13.782137 12002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45817
I20260926 13:16:13.782845 12002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:16:13.783295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 12002
I20260926 13:16:13.783409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:16:13.783587 392 external_mini_cluster.cc:1428] Setting key 2481b00bed2ddcf3a5aa64dcae54a34f
I20260926 13:16:13.787088 12130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45817 every 8 connection(s)
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:16:13.802241 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.788234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50151 (local address 127.0.98.62:39073)
0926 13:16:13.788276 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.788279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.788524 (+ 245us) server_negotiation.cc:407] Connection header received
0926 13:16:13.789232 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.789233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.789246 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.789278 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:13.790207 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.790978 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.792859 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.793171 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.793254 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.797067 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:13.797069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:13.797069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:13.797078 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:13.798801 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.799315 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.799317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.799317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.799381 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.801443 (+ 2062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.801444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.801445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.801575 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:13.801659 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.802035 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.802122 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:16:13.802448 12133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.788204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39073 (local address 127.0.98.3:50151)
0926 13:16:13.788382 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.788435 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.789105 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.789469 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.789473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.789758 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:16:13.790058 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.790066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.791510 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.791511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.792763 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.792765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.793279 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.793280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.793424 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.795462 (+ 2038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:16:13.795475 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:16:13.796946 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:16:13.798984 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:13.798989 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:13.798991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:13.799209 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:13.801219 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:13.801220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:13.801220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:13.801341 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:13.801650 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:16:13.801656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:16:13.801875 (+ 219us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.802000 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.802191 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260926 13:16:13.802903 12131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39073
I20260926 13:16:13.803040 12131 heartbeater.cc:464] Registering TS with master...
I20260926 13:16:13.803231 12131 heartbeater.cc:511] Master 127.0.98.62:39073 requested a full tablet report, sending...
I20260926 13:16:13.803632 11647 ts_manager.cc:194] Registered new tserver with Master: 29c83a6bee2747aebf11f550c22ea9ac (127.0.98.3:45817)
I20260926 13:16:13.804029 11647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50151
I20260926 13:16:13.805979 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:16:13.815900 11862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.807858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.98.62:39073)
0926 13:16:13.807902 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.807904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.808089 (+ 185us) server_negotiation.cc:407] Connection header received
0926 13:16:13.808163 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.808164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.808175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.808223 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:13.808969 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.809493 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.810276 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.810491 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.810584 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.811856 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:13.811857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:13.811858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:13.811864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:13.813306 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.813724 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.813725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.813726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.813783 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:13.814006 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:13.814007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:13.814007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:13.814129 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:13.814227 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.815775 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.815835 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:16:13.818058 11647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:16:13.819844 11647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:16:13.828579 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.823051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.0.1:46324)
0926 13:16:13.823700 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.823707 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.823732 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.824396 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.824397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.824399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:16:13.824452 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.824454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.825744 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.825745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.827217 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.827220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.828255 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.828257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.828362 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.828379 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.828430 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.828486 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":28,"thread_start_us":71,"threads_started":1}
I20260926 13:16:13.829315 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.823173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.0.98.3:45817)
0926 13:16:13.824234 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.824239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.824250 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:16:13.824287 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.824289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.824305 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.824522 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:16:13.824595 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.825644 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.827335 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.828144 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.829064 (+ 920us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.829158 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.829187 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.829230 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1009,"thread_start_us":98,"threads_started":1}
I20260926 13:16:13.830238 12065 tablet_service.cc:1467] Processing CreateTablet for tablet 1691a9c2efd048dfb288d59bffd8d0b5 (DEFAULT_TABLE table=test-table [id=cad1305ddd724535b42ee99b72fa40e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:16:13.830550 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.822748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35189 (local address 127.0.0.1:38278)
0926 13:16:13.823495 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.823539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.823605 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.824086 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.824089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.824091 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:16:13.824169 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.824171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.825793 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.825794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.829299 (+ 3505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.829302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.830310 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.830311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.830382 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.830386 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.830417 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.830476 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":80,"thread_start_us":83,"threads_started":1}
I20260926 13:16:13.830637 12065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1691a9c2efd048dfb288d59bffd8d0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:16:13.830876 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.822853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38278 (local address 127.0.98.2:35189)
0926 13:16:13.823906 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.823910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.823928 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:16:13.823967 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.823968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.823980 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.824082 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:16:13.824258 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.825713 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.829416 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.830207 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.830652 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.830735 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.830762 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.830802 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":98,"threads_started":1}
I20260926 13:16:13.833595 12149 tablet_bootstrap.cc:492] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Bootstrap starting.
I20260926 13:16:13.834524 11931 tablet_service.cc:1467] Processing CreateTablet for tablet 1691a9c2efd048dfb288d59bffd8d0b5 (DEFAULT_TABLE table=test-table [id=cad1305ddd724535b42ee99b72fa40e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:16:13.834692 12149 tablet_bootstrap.cc:654] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Neither blocks nor log segments found. Creating new log.
I20260926 13:16:13.834863 11931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1691a9c2efd048dfb288d59bffd8d0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:16:13.835539 12149 log.cc:824] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Log is configured to *not* fsync() on all Append() calls
I20260926 13:16:13.837881 12151 tablet_bootstrap.cc:492] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Bootstrap starting.
I20260926 13:16:13.838871 12151 tablet_bootstrap.cc:654] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Neither blocks nor log segments found. Creating new log.
I20260926 13:16:13.839165 12151 log.cc:824] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:16:13.840173 12151 tablet_bootstrap.cc:492] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: No bootstrap required, opened a new log
I20260926 13:16:13.840242 12151 ts_tablet_manager.cc:1397] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:16:13.842131 12151 raft_consensus.cc:348] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.842321 12151 raft_consensus.cc:374] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:16:13.842365 12151 raft_consensus.cc:729] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebe42baeb0a41ba9a1f75d74c44ccb8, State: Initialized, Role: FOLLOWER
I20260926 13:16:13.842489 12151 consensus_queue.cc:260] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.842751 12151 ts_tablet_manager.cc:1428] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 13:16:13.843344 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.823227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45849 (local address 127.0.0.1:46300)
0926 13:16:13.823787 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.823793 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.823816 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.838913 (+ 15097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.838914 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.838917 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:16:13.838987 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.838990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.840617 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.840618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.841773 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.841776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.843115 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.843116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.843180 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.843185 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.843218 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.843265 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":78,"threads_started":1}
I20260926 13:16:13.843876 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.823443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.98.1:45849)
0926 13:16:13.838719 (+ 15276us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.838724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.838737 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:16:13.838774 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.838776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.838793 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.839160 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:16:13.839281 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.840516 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.841881 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.843017 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.843671 (+ 654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.843742 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.843768 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.843807 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15236,"thread_start_us":67,"threads_started":1}
I20260926 13:16:13.844683 11793 tablet_service.cc:1467] Processing CreateTablet for tablet 1691a9c2efd048dfb288d59bffd8d0b5 (DEFAULT_TABLE table=test-table [id=cad1305ddd724535b42ee99b72fa40e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:16:13.845047 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1691a9c2efd048dfb288d59bffd8d0b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:16:13.846249 11997 heartbeater.cc:503] Master 127.0.98.62:39073 was elected leader, sending a full tablet report...
I20260926 13:16:13.848704 12154 tablet_bootstrap.cc:492] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f: Bootstrap starting.
I20260926 13:16:13.849610 12149 tablet_bootstrap.cc:492] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: No bootstrap required, opened a new log
I20260926 13:16:13.849711 12149 ts_tablet_manager.cc:1397] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260926 13:16:13.849893 12154 tablet_bootstrap.cc:654] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f: Neither blocks nor log segments found. Creating new log.
I20260926 13:16:13.850225 12154 log.cc:824] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:16:13.850919 12154 tablet_bootstrap.cc:492] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f: No bootstrap required, opened a new log
I20260926 13:16:13.850987 12154 ts_tablet_manager.cc:1397] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:16:13.851822 12149 raft_consensus.cc:348] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.852008 12149 raft_consensus.cc:374] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:16:13.852047 12149 raft_consensus.cc:729] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c83a6bee2747aebf11f550c22ea9ac, State: Initialized, Role: FOLLOWER
I20260926 13:16:13.852165 12149 consensus_queue.cc:260] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.852437 12149 ts_tablet_manager.cc:1428] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:16:13.852994 12131 heartbeater.cc:503] Master 127.0.98.62:39073 was elected leader, sending a full tablet report...
I20260926 13:16:13.853398 12154 raft_consensus.cc:348] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.853576 12154 raft_consensus.cc:374] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:16:13.853607 12154 raft_consensus.cc:729] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c1061774d54a359f4761c09b66c71f, State: Initialized, Role: FOLLOWER
I20260926 13:16:13.853940 12154 consensus_queue.cc:260] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.854245 12154 ts_tablet_manager.cc:1428] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:16:13.854383 11859 heartbeater.cc:503] Master 127.0.98.62:39073 was elected leader, sending a full tablet report...
Sep 26 13:16:13 dist-test-slave-f864 krb5kdc[11171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:13.989054 12153 raft_consensus.cc:482] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:16:13.989213 12153 raft_consensus.cc:504] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.989648 12153 leader_election.cc:288] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29c83a6bee2747aebf11f550c22ea9ac (127.0.98.3:45817), 93c1061774d54a359f4761c09b66c71f (127.0.98.1:45849)
I20260926 13:16:13.993521 11999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.989831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45817 (local address 127.0.98.2:56801)
0926 13:16:13.989889 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.989902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.989947 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.990204 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.990205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.990206 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:16:13.990293 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.990295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.991274 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.991275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.992495 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.992498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.993259 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.993260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.993333 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.993336 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.993384 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.993435 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":6}
I20260926 13:16:13.993554 12148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.990198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52061 (local address 127.0.98.1:45849)
0926 13:16:13.990314 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.990316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.990324 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:16:13.990349 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.990350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.990363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.990410 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:16:13.990598 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.991291 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.992616 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.993198 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.993235 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.993244 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.993452 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.993494 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260926 13:16:13.993820 12146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.989902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56801 (local address 127.0.98.3:45817)
0926 13:16:13.990031 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:13.990033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:13.990043 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:16:13.990074 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:13.990075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:13.990085 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:13.990132 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:16:13.990377 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.991167 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.992606 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.993149 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.993214 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.993227 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:13.993715 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 13:16:13.993762 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260926 13:16:13.994077 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1691a9c2efd048dfb288d59bffd8d0b5" candidate_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c83a6bee2747aebf11f550c22ea9ac" is_pre_election: true
I20260926 13:16:13.994251 12080 raft_consensus.cc:2454] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebe42baeb0a41ba9a1f75d74c44ccb8 in term 0.
I20260926 13:16:13.994267 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:13.989845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45849 (local address 127.0.98.2:52061)
0926 13:16:13.990030 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:13.990039 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:13.990083 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:13.990452 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:13.990453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:13.990454 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:16:13.990520 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.990521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.991383 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.991383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.992510 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:13.992512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:13.993294 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:13.993295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:13.993376 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:13.993380 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:16:13.994057 (+ 677us) client_negotiation.cc:240] Negotiation successful
0926 13:16:13.994105 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"thread_start_us":70,"threads_started":1}
I20260926 13:16:13.994506 11885 leader_election.cc:302] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c83a6bee2747aebf11f550c22ea9ac, 2ebe42baeb0a41ba9a1f75d74c44ccb8; no voters:
I20260926 13:16:13.994637 12153 raft_consensus.cc:2781] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:16:13.994690 12153 raft_consensus.cc:482] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:16:13.994721 12153 raft_consensus.cc:3037] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:16:13.994870 11813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1691a9c2efd048dfb288d59bffd8d0b5" candidate_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c1061774d54a359f4761c09b66c71f" is_pre_election: true
I20260926 13:16:13.995011 11813 raft_consensus.cc:2454] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ebe42baeb0a41ba9a1f75d74c44ccb8 in term 0.
I20260926 13:16:13.995577 12153 raft_consensus.cc:504] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.995733 12153 leader_election.cc:288] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [CANDIDATE]: Term 1 election: Requested vote from peers 29c83a6bee2747aebf11f550c22ea9ac (127.0.98.3:45817), 93c1061774d54a359f4761c09b66c71f (127.0.98.1:45849)
I20260926 13:16:13.995905 11813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1691a9c2efd048dfb288d59bffd8d0b5" candidate_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c1061774d54a359f4761c09b66c71f"
I20260926 13:16:13.995930 12080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1691a9c2efd048dfb288d59bffd8d0b5" candidate_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c83a6bee2747aebf11f550c22ea9ac"
I20260926 13:16:13.995992 12080 raft_consensus.cc:3037] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:16:13.996001 11813 raft_consensus.cc:3037] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:16:13.997251 11813 raft_consensus.cc:2454] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebe42baeb0a41ba9a1f75d74c44ccb8 in term 1.
I20260926 13:16:13.997251 12080 raft_consensus.cc:2454] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebe42baeb0a41ba9a1f75d74c44ccb8 in term 1.
I20260926 13:16:13.997428 11884 leader_election.cc:302] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ebe42baeb0a41ba9a1f75d74c44ccb8, 93c1061774d54a359f4761c09b66c71f; no voters:
I20260926 13:16:13.997557 12153 raft_consensus.cc:2781] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:16:13.997776 12153 raft_consensus.cc:686] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebe42baeb0a41ba9a1f75d74c44ccb8, State: Running, Role: LEADER
I20260926 13:16:13.997880 12153 consensus_queue.cc:237] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } }
I20260926 13:16:13.998780 11646 catalog_manager.cc:5495] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ebe42baeb0a41ba9a1f75d74c44ccb8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebe42baeb0a41ba9a1f75d74c44ccb8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 } health_report { overall_health: UNKNOWN } } }
W20260926 13:16:14.009935 11998 tablet.cc:2369] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:16:14.010625 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:14.008116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.0.98.2:35189)
0926 13:16:14.008163 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:14.008165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:14.008173 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:16:14.008251 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:14.008252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:14.008270 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:14.008332 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:16:14.008494 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:14.008949 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:14.009739 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:14.010018 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:14.010094 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:14.010314 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:16:14.010417 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:16:14.010512 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:14.010530 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 13:16:14.010550 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:16:14.012220 12080 raft_consensus.cc:1261] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 1 FOLLOWER]: Refusing update from remote peer 2ebe42baeb0a41ba9a1f75d74c44ccb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:16:14.012221 11813 raft_consensus.cc:1261] T 1691a9c2efd048dfb288d59bffd8d0b5 P 93c1061774d54a359f4761c09b66c71f [term 1 FOLLOWER]: Refusing update from remote peer 2ebe42baeb0a41ba9a1f75d74c44ccb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:16:14.012590 12159 consensus_queue.cc:1035] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c83a6bee2747aebf11f550c22ea9ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:16:14.012763 12159 consensus_queue.cc:1035] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c1061774d54a359f4761c09b66c71f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:16:14.014810 12162 mvcc.cc:205] Tried to move back new op lower bound from 7333595439151804416 to 7333595439096074240. Current Snapshot: MvccSnapshot[applied={T|T < 7333595439151804416}]
I20260926 13:16:14.015403 12163 mvcc.cc:205] Tried to move back new op lower bound from 7333595439151804416 to 7333595439096074240. Current Snapshot: MvccSnapshot[applied={T|T < 7333595439151804416}]
I20260926 13:16:14.015780 12164 mvcc.cc:205] Tried to move back new op lower bound from 7333595439151804416 to 7333595439096074240. Current Snapshot: MvccSnapshot[applied={T|T < 7333595439151804416}]
I20260926 13:16:14.018078 11646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:16:14.018209 11646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:16:14.018891 11646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 474cd5ac17ab485694ab77cdcc5757d7: Sending DeleteTablet for 3 replicas of tablet 1691a9c2efd048dfb288d59bffd8d0b5
I20260926 13:16:14.019397 11931 tablet_service.cc:1514] Processing DeleteTablet for tablet 1691a9c2efd048dfb288d59bffd8d0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:16:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38278
I20260926 13:16:14.019614 12065 tablet_service.cc:1514] Processing DeleteTablet for tablet 1691a9c2efd048dfb288d59bffd8d0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:16:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46324
I20260926 13:16:14.019825 11793 tablet_service.cc:1514] Processing DeleteTablet for tablet 1691a9c2efd048dfb288d59bffd8d0b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:16:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46300
I20260926 13:16:14.020195 12171 tablet_replica.cc:330] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: stopping tablet replica
I20260926 13:16:14.020290 12171 raft_consensus.cc:2229] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:16:14.020402 12171 raft_consensus.cc:2258] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:16:14.020464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 11728
I20260926 13:16:14.020543 12172 tablet_replica.cc:330] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: stopping tablet replica
I20260926 13:16:14.026330 12172 raft_consensus.cc:2229] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:16:14.026475 12172 raft_consensus.cc:2258] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:16:14.027017 12171 ts_tablet_manager.cc:1905] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:16:14.029018 12172 ts_tablet_manager.cc:1905] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:16:14.030644 12171 ts_tablet_manager.cc:1918] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:16:14.030748 12171 log.cc:1196] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/1691a9c2efd048dfb288d59bffd8d0b5
I20260926 13:16:14.031014 12171 ts_tablet_manager.cc:1939] T 1691a9c2efd048dfb288d59bffd8d0b5 P 2ebe42baeb0a41ba9a1f75d74c44ccb8: Deleting consensus metadata
I20260926 13:16:14.031481 11634 catalog_manager.cc:4841] TS 2ebe42baeb0a41ba9a1f75d74c44ccb8 (127.0.98.2:35189): tablet 1691a9c2efd048dfb288d59bffd8d0b5 (table test-table [id=cad1305ddd724535b42ee99b72fa40e4]) successfully deleted
I20260926 13:16:14.031683 12172 ts_tablet_manager.cc:1918] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:16:14.031756 12172 log.cc:1196] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/1691a9c2efd048dfb288d59bffd8d0b5
I20260926 13:16:14.032821 12172 ts_tablet_manager.cc:1939] T 1691a9c2efd048dfb288d59bffd8d0b5 P 29c83a6bee2747aebf11f550c22ea9ac: Deleting consensus metadata
I20260926 13:16:14.033250 11632 catalog_manager.cc:4841] TS 29c83a6bee2747aebf11f550c22ea9ac (127.0.98.3:45817): tablet 1691a9c2efd048dfb288d59bffd8d0b5 (table test-table [id=cad1305ddd724535b42ee99b72fa40e4]) successfully deleted
W20260926 13:16:14.034403 11631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45849 (error 108)
I20260926 13:16:14.034983 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 11866
I20260926 13:16:14.037209 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:14.037050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45849 (local address 127.0.0.1:46312)
0926 13:16:14.037116 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:14.037169 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 13:16:14.037428 11631 catalog_manager.cc:4584] TS 93c1061774d54a359f4761c09b66c71f (127.0.98.1:45849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1691a9c2efd048dfb288d59bffd8d0b5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45849: connect: Connection refused (error 111)
I20260926 13:16:14.040576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 12002
I20260926 13:16:14.046463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 11615
I20260926 13:16:14.051687 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:16:14.396745 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:16:14.396831 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:16:14.734654 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:16:14.734 UTC [11201] LOG: received smart shutdown request
2026-09-26 13:16:14.738 UTC [11201] LOG: background worker "logical replication launcher" (PID 11210) exited with exit code 1
2026-09-26 13:16:14.738 UTC [11205] LOG: shutting down
2026-09-26 13:16:14.920 UTC [11201] LOG: database system is shut down
2026-09-26T13:16:14Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46402 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:16:15 dist-test-slave-f864 krb5kdc[12183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:16:15 dist-test-slave-f864 krb5kdc[12183](info): set up 2 sockets
Sep 26 13:16:15 dist-test-slave-f864 krb5kdc[12183](info): commencing operation
krb5kdc: starting...
W20260926 13:16:16.058280 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:16:16 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:16:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 13:16:16.220536 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskLNUYBk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/postgres -l logfile start
2026-09-26 13:16:18.824 UTC [12213] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 13:16:18.824 UTC [12213] LOG: listening on IPv4 address "127.0.98.20", port 51925
2026-09-26 13:16:18.828 UTC [12213] LOG: listening on Unix socket "/tmp/.s.PGSQL.51925"
2026-09-26 13:16:18.863 UTC [12216] LOG: database system was shut down at 2026-09-26 13:16:17 UTC
2026-09-26 13:16:18.867 UTC [12213] LOG: database system is ready to accept connections
I20260926 13:16:19.801965 392 mini_postgres.cc:96] Postgres bound to 51925
2026-09-26 13:16:19.808 UTC [12224] FATAL: database "slave" does not exist
127.0.98.20:51925 - accepting connections
I20260926 13:16:19.808808 392 mini_ranger.cc:161] Starting Ranger...
I20260926 13:16:19.823525 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 13:16:20.334411 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 13:16:20.334501 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-admin
2026-09-26 13:16:20,609 [I] DB FLAVOR :POSTGRES
2026-09-26 13:16:20,610 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:16:20,610 [I] Checking connection..
2026-09-26 13:16:20,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 13:16:20,851 [I] Checking connection passed.
2026-09-26 13:16:20,851 [I] --------- Verifying version history table ---------
2026-09-26 13:16:20,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:16:21,083 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:16:21,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:16:21,318 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 13:16:21,318 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 13:16:21,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 13:16:21.549 UTC [12288] WARNING: there is no transaction in progress
2026-09-26 13:16:21,555 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 13:16:21,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 13:16:21,814 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 13:16:21,814 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:16:21,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 13:16:22,071 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 13:16:22,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-f864.c.gcp-upstream.internal','N') ;"
2026-09-26 13:16:22,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:16:22,813 [I] Table x_portal_user does not exist in database ranger
2026-09-26 13:16:22,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:16:23,041 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 13:16:23,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:16:23,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:16:23,515 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 13:16:23,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 13:16:24.485 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.501 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.517 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.532 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.556 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.576 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.629 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.636 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.650 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.659 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.669 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.683 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.692 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.699 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.705 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:24.712 UTC [12396] WARNING: there is no transaction in progress
2026-09-26 13:16:25,323 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 13:16:25,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 13:16:25,539 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 13:16:25,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 13:16:25,796 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 13:16:25,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 13:16:26,057 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 13:16:26,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 13:16:26,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 13:16:26,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-f864.c.gcp-upstream.internal';"
2026-09-26 13:16:26,771 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 13:16:26,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 13:16:27,025 [I] DB_PATCHES have already been applied
I20260926 13:16:27.031203 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskLNUYBk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 13:16:27.031266 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 13:16:27.034132 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34127
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 1:16:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 13:16:28.154078 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.345s user 0.000s sys 0.009s
Sep 26, 2026 1:16:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 1:16:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 1:16:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 1:16:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 1:16:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 13:16:50.103256 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 13:16:50 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428610, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 13:16:50.187211 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 13:16:50.196034 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 13:16:51.096247 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 13:16:51.096338 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms
I20260926 13:16:51.096356 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:51925
2026-09-26 13:16:51,255 [I] DB FLAVOR :POSTGRES
2026-09-26 13:16:51,255 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 13:16:51,255 [I] Checking connection
2026-09-26 13:16:51,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 13:16:51,481 [I] connection success
2026-09-26 13:16:51,481 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 13:16:51,482 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 13:16:51,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 13:16:51,724 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 13:16:51,725 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 13:16:51,725 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 13:16:51,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:51925/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 13:16:51,978 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 13:16:52.446941 392 mini_ranger_kms.cc:310] Created kms service
I20260926 13:16:52.651397 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 13:16:52.696995 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 13:16:52 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428610, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:53.101780 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 13:16:53.101902 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskLNUYBk/build/release/bin/postgresql.jar:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskLNUYBk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 13:16:53.101920 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 13:16:53.105476 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12576
I20260926 13:16:53.105569 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:38875
Sep 26, 2026 1:16:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 1:16:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 1:16:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 1:16:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 1:16:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 1:16:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 13:16:55.439916 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.253s user 0.006s sys 0.136s
I20260926 13:16:55.440021 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 13:16:55.440080 392 mini_ranger_kms.cc:401] 127.0.98.20:38875/kms/v1/keys
Sep 26 13:16:55 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 13:16:56 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:16:56 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428615, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 13:16:58 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:16:58.801126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43237
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43237
--ranger_config_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:16:58.896898 12627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:16:58.897131 12627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:16:58.897166 12627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:16:58.898514 12627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:16:58.898571 12627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:16:58.898604 12627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:16:58.898631 12627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:16:58.898659 12627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:16:58.900249 12627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:38875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43237
--ranger_config_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:16:58.900518 12627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:16:58.900749 12627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:16:58.906100 12635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:58.906129 12632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:58.906272 12627 server_base.cc:1034] running on GCE node
W20260926 13:16:58.906301 12633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:58.906646 12627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:16:58.906940 12627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:16:58.908129 12627 hybrid_clock.cc:648] HybridClock initialized: now 1790428618908109 us; error 38 us; skew 500 ppm
Sep 26 13:16:58 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:58.915907 12627 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:16:58.916441 12627 webserver.cc:460] Webserver started at http://127.0.98.62:38827/ using document root <none> and password file <none>
I20260926 13:16:58.916720 12627 fs_manager.cc:359] Metadata directory not provided
I20260926 13:16:58.916771 12627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:16:58.916903 12627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:16:58 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428618, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:58.967939 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "b6ee8af7cfb74235be12bce9605245a3"
format_stamp: "Formatted at 2026-09-26 13:16:58 on dist-test-slave-f864"
server_key: "42462a0f23832ea3655a4a67cbfe5e38"
server_key_iv: "52cd0a590d99386ef38fe46ac38d7e42"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:58.968389 12627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6ee8af7cfb74235be12bce9605245a3"
format_stamp: "Formatted at 2026-09-26 13:16:58 on dist-test-slave-f864"
server_key: "42462a0f23832ea3655a4a67cbfe5e38"
server_key_iv: "52cd0a590d99386ef38fe46ac38d7e42"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:58.969918 12627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:16:58.985879 12642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:58.986140 12627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:16:58.986289 12627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "b6ee8af7cfb74235be12bce9605245a3"
format_stamp: "Formatted at 2026-09-26 13:16:58 on dist-test-slave-f864"
server_key: "42462a0f23832ea3655a4a67cbfe5e38"
server_key_iv: "52cd0a590d99386ef38fe46ac38d7e42"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:58.986392 12627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:16:58.996624 12627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:16:58.998734 12627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:16:58.998891 12627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:16:59.004089 12627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43237
I20260926 13:16:59.004110 12694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43237 every 8 connection(s)
I20260926 13:16:59.004534 12627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:16:59.005225 12695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:16:59.007496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 12627
I20260926 13:16:59.007513 12695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Bootstrap starting.
I20260926 13:16:59.007588 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:16:59.007756 392 external_mini_cluster.cc:1428] Setting key 686c002509a904894f70604de1d47412
I20260926 13:16:59.008244 12695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Neither blocks nor log segments found. Creating new log.
I20260926 13:16:59.008615 12695 log.cc:824] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:16:59.009330 12695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: No bootstrap required, opened a new log
I20260926 13:16:59.011023 12695 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:16:59.011180 12695 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:16:59.011242 12695 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee8af7cfb74235be12bce9605245a3, State: Initialized, Role: FOLLOWER
I20260926 13:16:59.011376 12695 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:16:59.011471 12695 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:16:59.011524 12695 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:16:59.011601 12695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:16:59.012183 12695 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:16:59.012290 12695 leader_election.cc:302] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ee8af7cfb74235be12bce9605245a3; no voters:
I20260926 13:16:59.012449 12695 leader_election.cc:288] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:16:59.012501 12700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:16:59.012728 12700 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 LEADER]: Becoming Leader. State: Replica: b6ee8af7cfb74235be12bce9605245a3, State: Running, Role: LEADER
I20260926 13:16:59.012784 12695 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:16:59.012850 12700 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:16:59.013190 12700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6ee8af7cfb74235be12bce9605245a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } } }
I20260926 13:16:59.013259 12700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:16:59.013366 12700 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ee8af7cfb74235be12bce9605245a3. Latest consensus state: current_term: 1 leader_uuid: "b6ee8af7cfb74235be12bce9605245a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } } }
I20260926 13:16:59.013411 12700 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:16:59.013518 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:16:59.013983 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:16:59.014586 12695 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:16:59.016264 12704 catalog_manager.cc:1347] Generated new cluster ID: 81497d7d54cf4090abe62d96a3691494
I20260926 13:16:59.016309 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:16:59 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:16:59.025110 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:59.008272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.0.98.62:43237)
0926 13:16:59.008443 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:59.008452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:59.008485 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:16:59.009046 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:59.009073 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:59.009136 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:59.009285 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:59.010034 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:59.012023 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:59.013144 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:59.013814 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:59.013927 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:59.017053 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:59.017057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:59.017057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:59.017069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:59.023086 (+ 6017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:59.023518 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:59.023521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:59.023521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:59.023631 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:59.023802 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:59.023802 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:59.023802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:59.024111 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:59.024246 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:59.024718 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 13:16:59.024880 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260926 13:16:59.034837 12704 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:16:59.035636 12704 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:16:59.043690 12704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Generated new TSK 0
I20260926 13:16:59.043887 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:16:59.725673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:16:59.845885 12740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:16:59.846220 12740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:16:59.846258 12740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:16:59.848008 12740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:16:59.848073 12740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:16:59.848137 12740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:16:59.850348 12740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:38875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:16:59.850598 12740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:16:59.850862 12740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:16:59.856256 12748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:59.856349 12746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:16:59.856626 12745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:16:59.856889 12740 server_base.cc:1034] running on GCE node
I20260926 13:16:59.857100 12740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:16:59.857365 12740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:16:59.858538 12740 hybrid_clock.cc:648] HybridClock initialized: now 1790428619858526 us; error 28 us; skew 500 ppm
Sep 26 13:16:59 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:16:59.861171 12740 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:16:59.861585 12740 webserver.cc:460] Webserver started at http://127.0.98.1:34415/ using document root <none> and password file <none>
I20260926 13:16:59.861876 12740 fs_manager.cc:359] Metadata directory not provided
I20260926 13:16:59.861984 12740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:16:59.862141 12740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:16:59 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:16:59.875808 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a7d5394848c43f48e254174d7ea5ede"
format_stamp: "Formatted at 2026-09-26 13:16:59 on dist-test-slave-f864"
server_key: "6c84dac6bef06c0e889732220ccd347e"
server_key_iv: "ea9324356909a43c00899fea2a276246"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:59.876374 12740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a7d5394848c43f48e254174d7ea5ede"
format_stamp: "Formatted at 2026-09-26 13:16:59 on dist-test-slave-f864"
server_key: "6c84dac6bef06c0e889732220ccd347e"
server_key_iv: "ea9324356909a43c00899fea2a276246"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:59.878283 12740 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:16:59.896624 12755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:59.898459 12740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 13:16:59.898588 12740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "9a7d5394848c43f48e254174d7ea5ede"
format_stamp: "Formatted at 2026-09-26 13:16:59 on dist-test-slave-f864"
server_key: "6c84dac6bef06c0e889732220ccd347e"
server_key_iv: "ea9324356909a43c00899fea2a276246"
server_key_version: "kuduclusterkey@0"
I20260926 13:16:59.898703 12740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:16:59.917106 12740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:16:59.919677 12740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:16:59.919842 12740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:16:59.920105 12740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:16:59.920548 12740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:16:59.920640 12740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:59.920742 12740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:16:59.920814 12740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:16:59.928253 12740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45237
I20260926 13:16:59.928717 12740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:16:59.929201 12870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45237 every 8 connection(s)
I20260926 13:16:59.931998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 12740
I20260926 13:16:59.932086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:16:59.932258 392 external_mini_cluster.cc:1428] Setting key 46aef0ec94da4624a2bd180826e71e54
Sep 26 13:16:59 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428619, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:16:59.941139 12873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:59.929520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.1:36623)
0926 13:16:59.929704 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 13:16:59.929753 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:16:59.930560 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:16:59.931029 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:16:59.931034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:16:59.931286 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:16:59.931577 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:59.931587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:59.932560 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:59.932561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:59.934110 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:16:59.934113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:16:59.934702 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:16:59.934703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:16:59.934850 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:59.935544 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:16:59.935554 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:16:59.937233 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:16:59.939341 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:59.939348 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:59.939350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:59.939680 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:59.939908 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:16:59.939909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:16:59.939909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:16:59.940049 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:16:59.940377 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:16:59.940384 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:16:59.940587 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 13:16:59.940716 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 13:16:59.940883 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260926 13:16:59.941643 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:16:59.929514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36623 (local address 127.0.98.62:43237)
0926 13:16:59.930754 (+ 1240us) server_negotiation.cc:206] Beginning negotiation
0926 13:16:59.930756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:16:59.930765 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:16:59.930791 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:16:59.930792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:16:59.930804 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:16:59.930877 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:16:59.931705 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:59.932422 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:59.934267 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:16:59.934597 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:16:59.934668 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:16:59.937350 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:16:59.937352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:16:59.937352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:16:59.937360 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:16:59.939244 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:59.939772 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:59.939773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:59.939774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:59.939840 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:16:59.940130 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:16:59.940131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:16:59.940131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:16:59.940283 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:16:59.940358 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:16:59.941442 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0926 13:16:59.941551 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":105,"threads_started":1}
I20260926 13:16:59.942108 12871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 13:16:59.942248 12871 heartbeater.cc:464] Registering TS with master...
I20260926 13:16:59.942468 12871 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:16:59.943125 12657 ts_manager.cc:194] Registered new tserver with Master: 9a7d5394848c43f48e254174d7ea5ede (127.0.98.1:45237)
I20260926 13:16:59.944047 12657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36623
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:16:59.978646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:00.102140 12877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:00.102541 12877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:00.102626 12877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:00.104329 12877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:00.104440 12877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:00.104534 12877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:00.106909 12877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:38875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:00.107365 12877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:00.107676 12877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:00.114217 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.114598 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.114799 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:00.115820 12877 server_base.cc:1034] running on GCE node
I20260926 13:17:00.116122 12877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:00.116370 12877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:00.117542 12877 hybrid_clock.cc:648] HybridClock initialized: now 1790428620117519 us; error 34 us; skew 500 ppm
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:00.121004 12877 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:00.121543 12877 webserver.cc:460] Webserver started at http://127.0.98.2:37335/ using document root <none> and password file <none>
I20260926 13:17:00.121826 12877 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:00.121877 12877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:00.121996 12877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:00.138319 12877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "79a9da4fdd4b4743af2deed6c1774d77"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "46ee9365580f8ffb40100d858f455fcc"
server_key_iv: "cf87cea8147c4ac77199ee8e0737fdd0"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.138876 12877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79a9da4fdd4b4743af2deed6c1774d77"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "46ee9365580f8ffb40100d858f455fcc"
server_key_iv: "cf87cea8147c4ac77199ee8e0737fdd0"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.143038 12877 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260926 13:17:00.155789 12894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.156044 12877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:00.156137 12877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "79a9da4fdd4b4743af2deed6c1774d77"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "46ee9365580f8ffb40100d858f455fcc"
server_key_iv: "cf87cea8147c4ac77199ee8e0737fdd0"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.156229 12877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:00.180647 12877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:00.183142 12877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:00.183339 12877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:00.183585 12877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:00.183998 12877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:00.184092 12877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.184163 12877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:00.184219 12877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.191186 12877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42789
I20260926 13:17:00.191685 12877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:17:00.193445 13007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42789 every 8 connection(s)
I20260926 13:17:00.200987 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 12877
I20260926 13:17:00.201095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:17:00.201277 392 external_mini_cluster.cc:1428] Setting key 6cc4b94f7225a5d16a3a27afa56f75e6
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:00.205544 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:00.194418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40183 (local address 127.0.98.62:43237)
0926 13:17:00.194474 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:00.194476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:00.195322 (+ 846us) server_negotiation.cc:407] Connection header received
0926 13:17:00.196023 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:00.196024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:00.196039 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:00.196071 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:00.196978 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.197763 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.199278 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.199591 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.199665 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:00.202280 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:00.202282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:00.202282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:00.202289 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:00.204030 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:00.204498 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:00.204500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:00.204500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:00.204556 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:00.204814 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:00.204815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:00.204815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:00.204941 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:00.205004 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:00.205364 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 13:17:00.205449 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:17:00.206323 13010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:00.194138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.2:40183)
0926 13:17:00.195189 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:00.195235 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:00.195894 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:00.196278 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:00.196283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:00.196548 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:00.196827 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:00.196834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.197883 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.197884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:00.199161 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:00.199164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.199744 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.199745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:00.199884 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:00.200517 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:00.200527 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:00.202140 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:00.204128 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:00.204135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:00.204137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:00.204413 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:00.204621 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:00.204621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:00.204622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:00.204744 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:00.205037 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:00.205043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:00.205222 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 13:17:00.205860 (+ 638us) client_negotiation.cc:240] Negotiation successful
0926 13:17:00.206051 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":95,"threads_started":1}
I20260926 13:17:00.207124 13008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 13:17:00.207213 13008 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:00.207422 13008 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:00.207821 12657 ts_manager.cc:194] Registered new tserver with Master: 79a9da4fdd4b4743af2deed6c1774d77 (127.0.98.2:42789)
I20260926 13:17:00.208313 12657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40183
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:17:00.235610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:00.354893 13013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:00.355268 13013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:00.355350 13013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:00.357093 13013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:00.357327 13013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:00.357414 13013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:00.359611 13013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:38875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:00.360116 13013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:00.360441 13013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:00.366884 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.367291 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.367487 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:00.368680 13013 server_base.cc:1034] running on GCE node
I20260926 13:17:00.369091 13013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:00.369410 13013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:00.376806 13013 hybrid_clock.cc:648] HybridClock initialized: now 1790428620376783 us; error 40 us; skew 500 ppm
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:00.380502 13013 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:00.381079 13013 webserver.cc:460] Webserver started at http://127.0.98.3:39573/ using document root <none> and password file <none>
I20260926 13:17:00.381386 13013 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:00.381493 13013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:00.381675 13013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:00.402336 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "e457658f3796476e8b5ad081156b13cb"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "b6661a306cf97fcb18e1771b0cc5e1b2"
server_key_iv: "5dc0ae7bbcadc8222065aeab470366d5"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.403754 13013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e457658f3796476e8b5ad081156b13cb"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "b6661a306cf97fcb18e1771b0cc5e1b2"
server_key_iv: "5dc0ae7bbcadc8222065aeab470366d5"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.405548 13013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:00.426904 13029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.427155 13013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 13:17:00.427241 13013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "e457658f3796476e8b5ad081156b13cb"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "b6661a306cf97fcb18e1771b0cc5e1b2"
server_key_iv: "5dc0ae7bbcadc8222065aeab470366d5"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.427330 13013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:00.462143 13013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:00.464672 13013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:00.464884 13013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:00.465158 13013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:00.465596 13013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:00.465691 13013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.465760 13013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:00.465821 13013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.472141 13013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41139
I20260926 13:17:00.472671 13013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:17:00.475266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13013
I20260926 13:17:00.475354 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:17:00.475535 392 external_mini_cluster.cc:1428] Setting key 9c4c301a46d355e132cb5d3126efcb98
I20260926 13:17:00.478518 13142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41139 every 8 connection(s)
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:00.492461 12874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:00.479259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60725 (local address 127.0.98.62:43237)
0926 13:17:00.479305 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:00.479308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:00.481349 (+ 2041us) server_negotiation.cc:407] Connection header received
0926 13:17:00.482066 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:00.482067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:00.482080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:00.482110 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:00.482985 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.483710 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.485258 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.485556 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.485612 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:00.488982 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:00.488983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:00.488983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:00.488991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:00.490769 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:00.491281 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:00.491283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:00.491283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:00.491343 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:00.491650 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:00.491652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:00.491652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:00.491781 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:00.491872 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:00.492237 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 13:17:00.492355 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:17:00.492619 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:00.479254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.3:60725)
0926 13:17:00.481201 (+ 1947us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:00.481255 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:00.481934 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:00.482337 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:00.482341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:00.482604 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:00.482874 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:00.482881 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.483828 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.483829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:00.485153 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:00.485156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.485713 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.485714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:00.485853 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:00.486485 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:00.487133 (+ 648us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:00.488872 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:00.490898 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:00.490904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:00.490906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:00.491191 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:00.491416 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:00.491417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:00.491417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:00.491545 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:00.491868 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:00.491875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:00.492051 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 13:17:00.492170 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:17:00.492317 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"thread_start_us":93,"threads_started":1}
I20260926 13:17:00.493173 13143 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 13:17:00.493274 13143 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:00.493464 13143 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:00.493924 12657 ts_manager.cc:194] Registered new tserver with Master: e457658f3796476e8b5ad081156b13cb (127.0.98.3:41139)
I20260926 13:17:00.494521 12657 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60725
I20260926 13:17:00.497942 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:17:00.498695 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 12740
I20260926 13:17:00.504619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 12877
I20260926 13:17:00.509552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13013
I20260926 13:17:00.514745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 12627
I20260926 13:17:00.521301 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43237
--webserver_interface=127.0.98.62
--webserver_port=38827
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43237
--ranger_config_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:00.601397 13146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:00.601660 13146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:00.601696 13146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:00.602874 13146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:00.602946 13146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:00.602970 13146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:00.602993 13146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:00.603021 13146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:00.604631 13146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43237
--ranger_config_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:00.604908 13146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:00.605127 13146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:00.610380 13151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.610466 13152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:00.610589 13146 server_base.cc:1034] running on GCE node
W20260926 13:17:00.610381 13154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:00.610945 13146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:00.611326 13146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:00.612596 13146 hybrid_clock.cc:648] HybridClock initialized: now 1790428620612574 us; error 39 us; skew 500 ppm
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:00.615203 13146 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:00.615597 13146 webserver.cc:460] Webserver started at http://127.0.98.62:38827/ using document root <none> and password file <none>
I20260926 13:17:00.615835 13146 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:00.615888 13146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:00.681752 13146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:00.682561 13162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.682755 13146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:00.682868 13146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "b6ee8af7cfb74235be12bce9605245a3"
format_stamp: "Formatted at 2026-09-26 13:16:58 on dist-test-slave-f864"
server_key: "42462a0f23832ea3655a4a67cbfe5e38"
server_key_iv: "52cd0a590d99386ef38fe46ac38d7e42"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.683256 13146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:00.693778 13146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:00.695396 13146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:00.695561 13146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:00.699760 13146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43237
I20260926 13:17:00.699808 13214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43237 every 8 connection(s)
I20260926 13:17:00.700210 13146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:00.701476 13215 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:17:00.702071 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Bootstrap starting.
I20260926 13:17:00.707288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13146
I20260926 13:17:00.707621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45237
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43237
--webserver_port=34415
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:00.710422 13215 log.cc:824] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:00.712164 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:00.712462 13215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Bootstrap complete.
I20260926 13:17:00.714998 13215 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:17:00.715240 13215 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6ee8af7cfb74235be12bce9605245a3, State: Initialized, Role: FOLLOWER
I20260926 13:17:00.715411 13215 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:17:00.715483 13215 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:00.715553 13215 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:00.715605 13215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:17:00.716645 13215 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:17:00.716732 13215 leader_election.cc:302] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6ee8af7cfb74235be12bce9605245a3; no voters:
I20260926 13:17:00.716988 13215 leader_election.cc:288] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:17:00.717029 13219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:17:00.717146 13219 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [term 2 LEADER]: Becoming Leader. State: Replica: b6ee8af7cfb74235be12bce9605245a3, State: Running, Role: LEADER
I20260926 13:17:00.717263 13215 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:00.717296 13219 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } }
I20260926 13:17:00.717694 13221 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6ee8af7cfb74235be12bce9605245a3. Latest consensus state: current_term: 2 leader_uuid: "b6ee8af7cfb74235be12bce9605245a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } } }
I20260926 13:17:00.717775 13221 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:00.718110 13220 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6ee8af7cfb74235be12bce9605245a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6ee8af7cfb74235be12bce9605245a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43237 } } }
I20260926 13:17:00.718199 13220 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:00.718360 13225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:00.718936 13225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:00.719089 13225 catalog_manager.cc:1259] Loaded cluster ID: 81497d7d54cf4090abe62d96a3691494
I20260926 13:17:00.719172 13225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:17:00.719359 13215 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 13:17:00.722376 13225 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:00.723059 13225 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Loaded TSK: 0
I20260926 13:17:00.723392 13225 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:17:00.807632 13217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:00.808002 13217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:00.808097 13217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:00.810070 13217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:00.810146 13217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:00.810220 13217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:17:00.812806 13217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45237
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34415
--tserver_master_addrs=127.0.98.62:43237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:00.813084 13217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:00.813416 13217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:00.820184 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.820430 13255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:00.820173 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:00.821039 13217 server_base.cc:1034] running on GCE node
I20260926 13:17:00.821237 13217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:00.821516 13217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:00.822659 13217 hybrid_clock.cc:648] HybridClock initialized: now 1790428620822640 us; error 35 us; skew 500 ppm
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:00.827561 13217 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:17:00.828047 13217 webserver.cc:460] Webserver started at http://127.0.98.1:34415/ using document root <none> and password file <none>
I20260926 13:17:00.828305 13217 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:00.828385 13217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:00.848690 13217 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 13:17:00.849879 13263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.850131 13217 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:00.850266 13217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "9a7d5394848c43f48e254174d7ea5ede"
format_stamp: "Formatted at 2026-09-26 13:16:59 on dist-test-slave-f864"
server_key: "6c84dac6bef06c0e889732220ccd347e"
server_key_iv: "ea9324356909a43c00899fea2a276246"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:00.850749 13217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:00.913957 13217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:00.916589 13217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:00.916772 13217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:00.917027 13217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:00.917414 13217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:00.917476 13217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.917542 13217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:00.917585 13217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:00.925766 13217 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45237
I20260926 13:17:00.925834 13376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45237 every 8 connection(s)
I20260926 13:17:00.926409 13217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:17:00.929431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13217
I20260926 13:17:00.929721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42789
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43237
--webserver_port=37335
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:17:00 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428620, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:00.940927 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:00.927271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.1:59055)
0926 13:17:00.927818 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:00.927893 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:00.928412 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:00.929001 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:00.929005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:00.929285 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:00.929582 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:00.929589 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.931370 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.931371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:00.932690 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:00.932693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.933681 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.933682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:00.933829 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:00.934522 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:00.934533 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:00.936312 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:00.938811 (+ 2499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:00.938817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:00.938819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:00.939156 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:00.939466 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:00.939467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:00.939468 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:00.939605 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:00.940140 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:00.940146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:00.940345 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 13:17:00.940478 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 13:17:00.940647 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":124,"threads_started":1}
I20260926 13:17:00.941962 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:00.927344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59055 (local address 127.0.98.62:43237)
0926 13:17:00.928593 (+ 1249us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:00.928601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:00.928629 (+ 28us) server_negotiation.cc:407] Connection header received
0926 13:17:00.928753 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:00.928757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:00.928831 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:00.928962 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:00.929728 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.931238 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.932781 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:00.933455 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:00.933567 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:00.936460 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:00.936465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:00.936465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:00.936478 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:00.938714 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:00.939253 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:00.939257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:00.939257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:00.939384 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:00.939684 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:00.939685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:00.939685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:00.940020 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:00.940108 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:00.941596 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0926 13:17:00.941722 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":100,"threads_started":1}
I20260926 13:17:00.942843 13377 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 13:17:00.942974 13377 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:00.943153 13377 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
W20260926 13:17:00.944135 13377 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43237 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:17:00.944204 13377 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:00.944267 13377 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:00.944717 13377 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:00.944798 13377 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
W20260926 13:17:00.945140 13377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:17:01.041947 13381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:01.042279 13381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:01.042320 13381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:01.043767 13381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:01.043820 13381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:01.043851 13381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:01.045408 13381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37335
--tserver_master_addrs=127.0.98.62:43237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:01.045666 13381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:01.045928 13381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:01.052079 13387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:01.052079 13386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:01.052618 13381 server_base.cc:1034] running on GCE node
W20260926 13:17:01.052101 13389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:01.052954 13381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:01.053229 13381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:01.054425 13381 hybrid_clock.cc:648] HybridClock initialized: now 1790428621054407 us; error 30 us; skew 500 ppm
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:01.057257 13381 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:01.057730 13381 webserver.cc:460] Webserver started at http://127.0.98.2:37335/ using document root <none> and password file <none>
I20260926 13:17:01.057972 13381 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:01.058053 13381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:01.075178 13381 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 13:17:01.076020 13397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:01.076232 13381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:17:01.076313 13381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "79a9da4fdd4b4743af2deed6c1774d77"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "46ee9365580f8ffb40100d858f455fcc"
server_key_iv: "cf87cea8147c4ac77199ee8e0737fdd0"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:01.076751 13381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:01.112205 13381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:01.114374 13381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:01.114598 13381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:01.114810 13381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:01.115140 13381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:01.115195 13381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:01.115237 13381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:01.115265 13381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:01.122437 13381 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42789
I20260926 13:17:01.122949 13381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:17:01.123618 13510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42789 every 8 connection(s)
I20260926 13:17:01.126878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13381
I20260926 13:17:01.127157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:38875/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41139
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43237
--webserver_port=39573
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:01.135300 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:01.124573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56017 (local address 127.0.98.62:43237)
0926 13:17:01.125528 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:01.125530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:01.125792 (+ 262us) server_negotiation.cc:407] Connection header received
0926 13:17:01.126421 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:01.126422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:01.126441 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:01.126521 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:01.127377 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.128169 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.129750 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.129974 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.130068 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:01.132211 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:01.132213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:01.132213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:01.132221 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:01.133691 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:01.134219 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:01.134221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:01.134221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:01.134284 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:01.134609 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:01.134610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:01.134611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:01.134731 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:01.134806 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:01.135120 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 13:17:01.135210 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926}
I20260926 13:17:01.135546 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:01.124286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.2:56017)
0926 13:17:01.125602 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:01.125639 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:01.126298 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:01.126668 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:01.126672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:01.126945 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:01.127239 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:01.127246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.128303 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.128305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:01.129644 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:01.129647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.130094 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.130096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:01.130268 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:01.130812 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:01.130821 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:01.132109 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:01.133782 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:01.133788 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:01.133790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:01.134088 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:01.134360 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:01.134361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:01.134361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:01.134490 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:01.134797 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:01.134803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:01.134976 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 13:17:01.135089 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:17:01.135237 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":87,"threads_started":1}
I20260926 13:17:01.136054 13511 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 13:17:01.136202 13511 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.136438 13511 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
W20260926 13:17:01.137107 13511 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43237 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:17:01.137233 13511 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.137331 13511 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:01.137637 13511 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.137744 13511 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
W20260926 13:17:01.138044 13511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:17:01.248173 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:01.248435 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:01.248473 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:01.250245 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:01.250303 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:01.250341 13514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:01.252523 13514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:38875/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39573
--tserver_master_addrs=127.0.98.62:43237
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:01.252781 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:01.253089 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:01.259631 13520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:01.260032 13521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:01.260514 13523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:01.261891 13514 server_base.cc:1034] running on GCE node
I20260926 13:17:01.262147 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:01.262509 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:01.263706 13514 hybrid_clock.cc:648] HybridClock initialized: now 1790428621263679 us; error 46 us; skew 500 ppm
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:01.266393 13514 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:01.266906 13514 webserver.cc:460] Webserver started at http://127.0.98.3:39573/ using document root <none> and password file <none>
I20260926 13:17:01.267159 13514 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:01.267216 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:01.284998 13514 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:01.285887 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:01.286094 13514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:01.286229 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "e457658f3796476e8b5ad081156b13cb"
format_stamp: "Formatted at 2026-09-26 13:17:00 on dist-test-slave-f864"
server_key: "b6661a306cf97fcb18e1771b0cc5e1b2"
server_key_iv: "5dc0ae7bbcadc8222065aeab470366d5"
server_key_version: "kuduclusterkey@0"
I20260926 13:17:01.289665 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:01.311769 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:01.314273 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:01.314481 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:01.314766 13514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:01.315193 13514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:01.315248 13514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:01.315289 13514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:01.315313 13514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:01.321897 13514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41139
I20260926 13:17:01.323613 13644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41139 every 8 connection(s)
I20260926 13:17:01.323997 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:01.333765 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:01.324795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47589 (local address 127.0.98.62:43237)
0926 13:17:01.324871 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:01.324873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:01.325119 (+ 246us) server_negotiation.cc:407] Connection header received
0926 13:17:01.325656 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:01.325657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:01.325672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:01.325753 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:01.326461 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.327211 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.328221 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.328554 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.328641 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:01.330918 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:01.330919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:01.330920 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:01.330927 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:01.332330 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:01.332763 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:01.332764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:01.332765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:01.332821 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:01.333078 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:01.333079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:01.333079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:01.333201 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:01.333276 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:01.333583 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 13:17:01.333672 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:17:01.333967 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:01.324775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43237 (local address 127.0.98.3:47589)
0926 13:17:01.324956 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:01.325000 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:01.325505 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:01.325826 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:01.325830 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:01.326094 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:01.326365 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:01.326371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.327303 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.327303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:01.328130 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:01.328132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.328671 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.328672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:01.328817 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:01.329378 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:01.329387 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:01.330824 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:01.332434 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:01.332442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:01.332444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:01.332672 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:01.332896 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:01.332897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:01.332897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:01.333009 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:01.333269 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:01.333275 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:01.333432 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 13:17:01.333567 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 13:17:01.333719 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
I20260926 13:17:01.334336 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13514
I20260926 13:17:01.334983 13645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43237
I20260926 13:17:01.335083 13645 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.335263 13645 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
W20260926 13:17:01.335944 13645 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43237 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 13:17:01.336030 13645 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.336094 13645 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:01.336404 13645 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.336480 13645 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
W20260926 13:17:01.336800 13645 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:17:01.346491 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:01.334780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.0.98.62:43237)
0926 13:17:01.334931 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:01.334933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:01.335780 (+ 847us) server_negotiation.cc:407] Connection header received
0926 13:17:01.336266 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:01.336267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:01.336278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:01.336307 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:01.337133 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.337596 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.342133 (+ 4537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:01.342411 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:01.342501 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:01.343798 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:01.343799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:01.343799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:01.343807 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:01.345246 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:01.345634 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:01.345635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:01.345636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:01.345694 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:01.345899 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:01.345900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:01.345900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:01.346010 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:01.346278 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:01.346316 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 13:17:01.346385 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":59,"threads_started":1}
Sep 26 13:17:01 dist-test-slave-f864 krb5kdc[12183](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:01.945338 13377 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:01.945495 13377 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:01.945905 13179 ts_manager.cc:194] Registered new tserver with Master: 9a7d5394848c43f48e254174d7ea5ede (127.0.98.1:45237)
I20260926 13:17:01.946974 13179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59055
I20260926 13:17:02.138239 13511 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:02.138397 13511 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:02.138881 13179 ts_manager.cc:194] Registered new tserver with Master: 79a9da4fdd4b4743af2deed6c1774d77 (127.0.98.2:42789)
I20260926 13:17:02.139411 13179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56017
Sep 26 13:17:02 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 13:17:02.336989 13645 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:02.337142 13645 heartbeater.cc:511] Master 127.0.98.62:43237 requested a full tablet report, sending...
I20260926 13:17:02.337492 13179 ts_manager.cc:194] Registered new tserver with Master: e457658f3796476e8b5ad081156b13cb (127.0.98.3:41139)
I20260926 13:17:02.337997 13179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47589
I20260926 13:17:02.343015 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:17:02.352278 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.344077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46300 (local address 127.0.98.62:43237)
0926 13:17:02.344231 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.344233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.344400 (+ 167us) server_negotiation.cc:407] Connection header received
0926 13:17:02.344423 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.344424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.344435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.344495 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:02.345174 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.345903 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.346898 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.347218 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.347281 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.348887 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:02.348888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:02.348889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:02.348897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:02.350778 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:02.351161 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:02.351163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:02.351163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:02.351223 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:02.351412 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:02.351412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:02.351413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:02.351553 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:02.351654 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.352087 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.352170 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
I20260926 13:17:02.354768 13179 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:17:02.357630 13179 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:17:02.373462 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.364876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45237 (local address 127.0.0.1:48440)
0926 13:17:02.369348 (+ 4472us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.369359 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:02.369403 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:02.369702 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:02.369703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:02.369705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:02.369744 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.369745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.370989 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.370990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.372102 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.372105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.373140 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.373141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.373227 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.373241 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:17:02.373313 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:17:02.373365 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4454,"thread_start_us":65,"threads_started":1}
I20260926 13:17:02.373936 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.365536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48440 (local address 127.0.98.1:45237)
0926 13:17:02.366428 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.366432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.369465 (+ 3033us) server_negotiation.cc:407] Connection header received
0926 13:17:02.369505 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.369508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.369525 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.369598 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:02.369820 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.370886 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.372201 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.372943 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.373038 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.373103 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.373810 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.373856 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":49,"threads_started":1}
I20260926 13:17:02.374936 13311 tablet_service.cc:1467] Processing CreateTablet for tablet d3f3a456bd304a7d9b61e38274701df2 (DEFAULT_TABLE table=test-table [id=ab453e333896424cb11fefacdbdad781]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:02.375330 13311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f3a456bd304a7d9b61e38274701df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:02.379807 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.364646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41139 (local address 127.0.0.1:54564)
0926 13:17:02.365860 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.365903 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:02.365973 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:02.366542 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:02.366546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:02.366548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:02.366617 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.366619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.367889 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.367890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.369227 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.369230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.379547 (+ 10317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.379548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.379618 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.379623 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:17:02.379680 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:17:02.379728 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":64,"threads_started":1}
I20260926 13:17:02.379825 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.365386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.0.98.3:41139)
0926 13:17:02.366140 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.366145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.366156 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:02.366210 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.366212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.366228 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.366309 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:02.366712 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.367788 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.374452 (+ 6664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.375183 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.375266 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.375322 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.379700 (+ 4378us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.379751 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":61,"threads_started":1}
I20260926 13:17:02.380885 13574 tablet_service.cc:1467] Processing CreateTablet for tablet d3f3a456bd304a7d9b61e38274701df2 (DEFAULT_TABLE table=test-table [id=ab453e333896424cb11fefacdbdad781]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:02.381230 13574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f3a456bd304a7d9b61e38274701df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:02.383371 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.365039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42789 (local address 127.0.0.1:55276)
0926 13:17:02.374311 (+ 9272us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.374322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:02.374379 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:02.379456 (+ 5077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:02.379457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:02.379459 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:02.379514 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.379516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.381082 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.381083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.382196 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.382199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.383138 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.383139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.383199 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.383204 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:17:02.383232 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 13:17:02.383270 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9252,"thread_start_us":65,"threads_started":1}
I20260926 13:17:02.383706 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.365676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55276 (local address 127.0.98.2:42789)
0926 13:17:02.366497 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.366500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.379301 (+ 12801us) server_negotiation.cc:407] Connection header received
0926 13:17:02.379344 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.379346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.379366 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.379458 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:02.379608 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.380977 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.382334 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.383034 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.383466 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.383560 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.383587 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.383635 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":48,"threads_started":1}
I20260926 13:17:02.385480 13679 tablet_bootstrap.cc:492] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Bootstrap starting.
I20260926 13:17:02.385601 13445 tablet_service.cc:1467] Processing CreateTablet for tablet d3f3a456bd304a7d9b61e38274701df2 (DEFAULT_TABLE table=test-table [id=ab453e333896424cb11fefacdbdad781]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:02.385932 13445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f3a456bd304a7d9b61e38274701df2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:02.386359 13679 tablet_bootstrap.cc:654] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:02.386715 13679 log.cc:824] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:02.387128 13678 tablet_bootstrap.cc:492] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: Bootstrap starting.
I20260926 13:17:02.387805 13678 tablet_bootstrap.cc:654] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:02.388223 13678 log.cc:824] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:02.388979 13679 tablet_bootstrap.cc:492] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: No bootstrap required, opened a new log
I20260926 13:17:02.389108 13679 ts_tablet_manager.cc:1397] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:17:02.396492 13679 raft_consensus.cc:348] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.396893 13679 raft_consensus.cc:374] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:02.397003 13679 raft_consensus.cc:729] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e457658f3796476e8b5ad081156b13cb, State: Initialized, Role: FOLLOWER
I20260926 13:17:02.397260 13679 consensus_queue.cc:260] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.397663 13679 ts_tablet_manager.cc:1428] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20260926 13:17:02.397830 13645 heartbeater.cc:503] Master 127.0.98.62:43237 was elected leader, sending a full tablet report...
I20260926 13:17:02.399813 13678 tablet_bootstrap.cc:492] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: No bootstrap required, opened a new log
I20260926 13:17:02.399948 13678 ts_tablet_manager.cc:1397] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260926 13:17:02.402755 13678 raft_consensus.cc:348] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.403002 13678 raft_consensus.cc:374] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:02.403108 13678 raft_consensus.cc:729] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a7d5394848c43f48e254174d7ea5ede, State: Initialized, Role: FOLLOWER
I20260926 13:17:02.403317 13678 consensus_queue.cc:260] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.403676 13678 ts_tablet_manager.cc:1428] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:17:02.403867 13377 heartbeater.cc:503] Master 127.0.98.62:43237 was elected leader, sending a full tablet report...
I20260926 13:17:02.404517 13682 tablet_bootstrap.cc:492] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Bootstrap starting.
I20260926 13:17:02.405368 13682 tablet_bootstrap.cc:654] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:02.405755 13682 log.cc:824] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:02.406653 13682 tablet_bootstrap.cc:492] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: No bootstrap required, opened a new log
I20260926 13:17:02.406723 13682 ts_tablet_manager.cc:1397] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:02.409399 13682 raft_consensus.cc:348] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.409552 13682 raft_consensus.cc:374] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:02.409585 13682 raft_consensus.cc:729] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a9da4fdd4b4743af2deed6c1774d77, State: Initialized, Role: FOLLOWER
I20260926 13:17:02.410240 13682 consensus_queue.cc:260] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.410579 13682 ts_tablet_manager.cc:1428] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:17:02.410737 13511 heartbeater.cc:503] Master 127.0.98.62:43237 was elected leader, sending a full tablet report...
W20260926 13:17:02.428591 13378 tablet.cc:2369] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 13:17:02 dist-test-slave-f864 krb5kdc[12183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 13:17:02.575284 13646 tablet.cc:2369] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:17:02.624496 13512 tablet.cc:2369] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:17:02.650068 13684 raft_consensus.cc:482] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:17:02.650264 13684 raft_consensus.cc:504] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.650707 13684 leader_election.cc:288] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e457658f3796476e8b5ad081156b13cb (127.0.98.3:41139), 79a9da4fdd4b4743af2deed6c1774d77 (127.0.98.2:42789)
I20260926 13:17:02.655721 13677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.651612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33545 (local address 127.0.98.2:42789)
0926 13:17:02.651687 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.651689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.651758 (+ 69us) server_negotiation.cc:407] Connection header received
0926 13:17:02.651892 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.651893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.651905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.651939 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:02.652197 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.652975 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.654598 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.655231 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.655285 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.655298 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.655592 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.655639 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:17:02.656260 13689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.651403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42789 (local address 127.0.98.1:33545)
0926 13:17:02.651689 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.651700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:02.651788 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:02.652010 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:02.652011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:02.652012 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:02.652112 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.652115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.653091 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.653092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.654469 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.654472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.655394 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.655395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.655502 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.655506 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:02.656094 (+ 588us) client_negotiation.cc:240] Negotiation successful
0926 13:17:02.656146 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":98,"threads_started":1}
I20260926 13:17:02.656809 13456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f3a456bd304a7d9b61e38274701df2" candidate_uuid: "9a7d5394848c43f48e254174d7ea5ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a9da4fdd4b4743af2deed6c1774d77" is_pre_election: true
I20260926 13:17:02.656988 13456 raft_consensus.cc:2454] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7d5394848c43f48e254174d7ea5ede in term 0.
I20260926 13:17:02.657285 13267 leader_election.cc:302] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79a9da4fdd4b4743af2deed6c1774d77, 9a7d5394848c43f48e254174d7ea5ede; no voters:
I20260926 13:17:02.657433 13684 raft_consensus.cc:2781] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:17:02.657476 13684 raft_consensus.cc:482] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:17:02.657501 13684 raft_consensus.cc:3037] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:02.658514 13684 raft_consensus.cc:504] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.658666 13684 leader_election.cc:288] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [CANDIDATE]: Term 1 election: Requested vote from peers e457658f3796476e8b5ad081156b13cb (127.0.98.3:41139), 79a9da4fdd4b4743af2deed6c1774d77 (127.0.98.2:42789)
I20260926 13:17:02.659101 13456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f3a456bd304a7d9b61e38274701df2" candidate_uuid: "9a7d5394848c43f48e254174d7ea5ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a9da4fdd4b4743af2deed6c1774d77"
I20260926 13:17:02.659196 13456 raft_consensus.cc:3037] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:02.660158 13456 raft_consensus.cc:2454] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7d5394848c43f48e254174d7ea5ede in term 1.
I20260926 13:17:02.660434 13267 leader_election.cc:302] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79a9da4fdd4b4743af2deed6c1774d77, 9a7d5394848c43f48e254174d7ea5ede; no voters:
I20260926 13:17:02.660571 13684 raft_consensus.cc:2781] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:02.660733 13684 raft_consensus.cc:686] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [term 1 LEADER]: Becoming Leader. State: Replica: 9a7d5394848c43f48e254174d7ea5ede, State: Running, Role: LEADER
I20260926 13:17:02.660838 13684 consensus_queue.cc:237] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } }
I20260926 13:17:02.661724 13179 catalog_manager.cc:5495] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a7d5394848c43f48e254174d7ea5ede (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9a7d5394848c43f48e254174d7ea5ede" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a7d5394848c43f48e254174d7ea5ede" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 } health_report { overall_health: UNKNOWN } } }
I20260926 13:17:02.663184 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.650914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41139 (local address 127.0.98.1:58035)
0926 13:17:02.651583 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.651596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:02.651742 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:02.651941 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:02.651942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:02.651944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:02.652259 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.652262 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.659114 (+ 6852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.659114 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.660325 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:02.660329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.662917 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.662918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:02.662992 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.662995 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:17:02.663061 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:17:02.663103 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":118,"threads_started":1}
I20260926 13:17:02.663231 13675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.651179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58035 (local address 127.0.98.3:41139)
0926 13:17:02.651240 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.651243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.651710 (+ 467us) server_negotiation.cc:407] Connection header received
0926 13:17:02.651831 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.651832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.651844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.651874 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:02.658264 (+ 6390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.658983 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.662186 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.662771 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.662824 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.662836 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.663097 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.663154 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:17:02.663771 13599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f3a456bd304a7d9b61e38274701df2" candidate_uuid: "9a7d5394848c43f48e254174d7ea5ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e457658f3796476e8b5ad081156b13cb" is_pre_election: true
I20260926 13:17:02.663932 13599 raft_consensus.cc:2454] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a7d5394848c43f48e254174d7ea5ede in term 0.
I20260926 13:17:02.664079 13598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3f3a456bd304a7d9b61e38274701df2" candidate_uuid: "9a7d5394848c43f48e254174d7ea5ede" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e457658f3796476e8b5ad081156b13cb"
I20260926 13:17:02.664145 13598 raft_consensus.cc:3037] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:02.665007 13598 raft_consensus.cc:2454] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a7d5394848c43f48e254174d7ea5ede in term 1.
I20260926 13:17:02.721217 13676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.717825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.0.98.1:45237)
0926 13:17:02.717871 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:02.717873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:02.717884 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:02.717969 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:02.717970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:02.717996 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:02.718073 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:02.718332 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.719098 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.720259 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:02.720559 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:02.720642 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:02.720840 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:02.720977 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:17:02.721091 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:02.721110 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 13:17:02.721136 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:17:02.722993 13456 raft_consensus.cc:1261] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 1 FOLLOWER]: Refusing update from remote peer 9a7d5394848c43f48e254174d7ea5ede: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:02.723068 13598 raft_consensus.cc:1261] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 1 FOLLOWER]: Refusing update from remote peer 9a7d5394848c43f48e254174d7ea5ede: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:02.723377 13684 consensus_queue.cc:1035] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a9da4fdd4b4743af2deed6c1774d77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:02.723556 13684 consensus_queue.cc:1035] T d3f3a456bd304a7d9b61e38274701df2 P 9a7d5394848c43f48e254174d7ea5ede [LEADER]: Connected to new peer: Peer: permanent_uuid: "e457658f3796476e8b5ad081156b13cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:02.726186 13696 mvcc.cc:205] Tried to move back new op lower bound from 7333595638670733312 to 7333595638419423232. Current Snapshot: MvccSnapshot[applied={T|T < 7333595638670733312}]
I20260926 13:17:02.726828 13700 mvcc.cc:205] Tried to move back new op lower bound from 7333595638670733312 to 7333595638419423232. Current Snapshot: MvccSnapshot[applied={T|T < 7333595638670733312}]
I20260926 13:17:02.729097 13179 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:02.729182 13179 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:02.729899 13179 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6ee8af7cfb74235be12bce9605245a3: Sending DeleteTablet for 3 replicas of tablet d3f3a456bd304a7d9b61e38274701df2
I20260926 13:17:02.730309 13445 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f3a456bd304a7d9b61e38274701df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55276
I20260926 13:17:02.730413 13574 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f3a456bd304a7d9b61e38274701df2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54564
I20260926 13:17:02.730646 13707 tablet_replica.cc:330] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: stopping tablet replica
I20260926 13:17:02.730796 13707 raft_consensus.cc:2229] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:02.730885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13217
I20260926 13:17:02.730955 13707 raft_consensus.cc:2258] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:02.732450 13706 tablet_replica.cc:330] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: stopping tablet replica
I20260926 13:17:02.732564 13706 raft_consensus.cc:2229] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:02.732677 13706 raft_consensus.cc:2258] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:02.736110 13707 ts_tablet_manager.cc:1905] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:17:02.736801 13166 connection.cc:535] client connection to 127.0.98.1:45237 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 13:17:02.736899 13166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:17:02.737142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13381
I20260926 13:17:02.738698 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.738569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45237 (local address 127.0.0.1:48456)
0926 13:17:02.738621 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.738655 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:17:02.738907 13166 catalog_manager.cc:4584] TS 9a7d5394848c43f48e254174d7ea5ede (127.0.98.1:45237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3f3a456bd304a7d9b61e38274701df2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45237: connect: Connection refused (error 111)
I20260926 13:17:02.737183 13706 ts_tablet_manager.cc:1905] T d3f3a456bd304a7d9b61e38274701df2 P 79a9da4fdd4b4743af2deed6c1774d77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:02.739213 13707 ts_tablet_manager.cc:1918] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:02.739306 13707 log.cc:1196] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/d3f3a456bd304a7d9b61e38274701df2
I20260926 13:17:02.739579 13707 ts_tablet_manager.cc:1939] T d3f3a456bd304a7d9b61e38274701df2 P e457658f3796476e8b5ad081156b13cb: Deleting consensus metadata
I20260926 13:17:02.740263 13165 catalog_manager.cc:4841] TS e457658f3796476e8b5ad081156b13cb (127.0.98.3:41139): tablet d3f3a456bd304a7d9b61e38274701df2 (table test-table [id=ab453e333896424cb11fefacdbdad781]) successfully deleted
I20260926 13:17:02.742518 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:02.742382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42789 (local address 127.0.0.1:55292)
0926 13:17:02.742431 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:02.742458 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:17:02.742552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13514
I20260926 13:17:02.747587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13146
I20260926 13:17:02.754698 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 13:17:03.100559 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 13:17:03.100906 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 13:17:03.438156 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 13:17:03.438 UTC [12213] LOG: received smart shutdown request
2026-09-26 13:17:03.442 UTC [12213] LOG: background worker "logical replication launcher" (PID 12222) exited with exit code 1
2026-09-26 13:17:03.442 UTC [12217] LOG: shutting down
2026-09-26 13:17:03.616 UTC [12213] LOG: database system is shut down
2026-09-26T13:17:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48695 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:17:03 dist-test-slave-f864 krb5kdc[13721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:17:03 dist-test-slave-f864 krb5kdc[13721](info): set up 2 sockets
Sep 26 13:17:03 dist-test-slave-f864 krb5kdc[13721](info): commencing operation
krb5kdc: starting...
W20260926 13:17:04.778106 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:17:04 dist-test-slave-f864 krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:17:04.897547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38071
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38071,127.0.98.61:42417,127.0.98.60:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:04.978735 13734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:04.978940 13734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:04.978962 13734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:04.980118 13734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:04.980173 13734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:04.980188 13734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:04.980201 13734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:04.980252 13734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:04.981786 13734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38071,127.0.98.61:42417,127.0.98.60:45251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:04.982038 13734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:04.982301 13734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:04.987057 13739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:04.987088 13742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:04.987097 13740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:04.987260 13734 server_base.cc:1034] running on GCE node
I20260926 13:17:04.987498 13734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:04.987828 13734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:04.989008 13734 hybrid_clock.cc:648] HybridClock initialized: now 1790428624988990 us; error 45 us; skew 500 ppm
Sep 26 13:17:04 dist-test-slave-f864 krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:04.998116 13734 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:04.998642 13734 webserver.cc:460] Webserver started at http://127.0.98.62:38809/ using document root <none> and password file <none>
I20260926 13:17:04.998890 13734 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:04.998960 13734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:04.999104 13734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:05.000161 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "ac3d481e2c02439181124b55b6636bfd"
format_stamp: "Formatted at 2026-09-26 13:17:04 on dist-test-slave-f864"
server_key: "084b25742115cf22b53e00c89aa652f2"
server_key_iv: "270a25d92ca4281ddcd0a071b0a69b11"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.000576 13734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "ac3d481e2c02439181124b55b6636bfd"
format_stamp: "Formatted at 2026-09-26 13:17:04 on dist-test-slave-f864"
server_key: "084b25742115cf22b53e00c89aa652f2"
server_key_iv: "270a25d92ca4281ddcd0a071b0a69b11"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.001962 13734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260926 13:17:05.002836 13749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:05.003036 13734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:17:05.003142 13734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "ac3d481e2c02439181124b55b6636bfd"
format_stamp: "Formatted at 2026-09-26 13:17:04 on dist-test-slave-f864"
server_key: "084b25742115cf22b53e00c89aa652f2"
server_key_iv: "270a25d92ca4281ddcd0a071b0a69b11"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.003228 13734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:05.015276 13734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:05.017573 13734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:05.017721 13734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:05.021628 13734 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38071
I20260926 13:17:05.021673 13801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38071 every 8 connection(s)
I20260926 13:17:05.022109 13734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:05.022256 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13734
I20260926 13:17:05.022348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:17:05.022573 392 external_mini_cluster.cc:1428] Setting key 22610f5e0b3fe5089f142ae2b08c78d8
I20260926 13:17:05.023106 13802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:05.025333 13802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:05.037269 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.026225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.0.1:43752)
0926 13:17:05.026401 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.026446 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.026881 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.027232 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.027235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.027620 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.027857 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.027863 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.028901 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.028902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.030068 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.030071 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.030608 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.030609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.030668 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.031185 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.031193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.033615 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.035787 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.035793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.035794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.035985 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.036250 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.036251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.036251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.036372 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.036844 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.036849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.036904 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.036982 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.037108 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260926 13:17:05.037384 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.026284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43752 (local address 127.0.98.62:38071)
0926 13:17:05.027017 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.027021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.027032 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:05.027091 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.027092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.027107 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.027169 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.027961 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.028803 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.030184 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.030529 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.030609 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.033696 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.033703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.033703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.033709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.035696 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.036067 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.036070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.036071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.036178 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.036447 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.036448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.036448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.036770 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.036864 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.037042 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.037196 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":704,"thread_start_us":115,"threads_started":1}
I20260926 13:17:05.038120 13802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:17:05.038507 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.038389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34086)
0926 13:17:05.038434 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.038479 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:17:05.038664 13751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
I20260926 13:17:05.039064 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.038991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34100)
0926 13:17:05.039025 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.039049 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 13:17:05.039181 13802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42417: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 13:17:05.049752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42417
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38071,127.0.98.61:42417,127.0.98.60:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:05.088405 13802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } attempt: 1
I20260926 13:17:05.088924 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.088794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34108)
0926 13:17:05.088857 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.088896 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:17:05.089257 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.089168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34120)
0926 13:17:05.089212 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.089236 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:17:05.089359 13802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42417: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
W20260926 13:17:05.136960 13809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:05.137207 13809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:05.137243 13809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:05.138419 13809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:05.138487 13809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:05.138510 13809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:05.138530 13809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:05.138561 13809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:05.140065 13809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38071,127.0.98.61:42417,127.0.98.60:45251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:05.140347 13809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:05.140579 13809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:05.145591 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:05.145618 13809 server_base.cc:1034] running on GCE node
W20260926 13:17:05.145637 13817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:05.145588 13815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:05.146100 13809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:05.146435 13809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:05.147598 13809 hybrid_clock.cc:648] HybridClock initialized: now 1790428625147590 us; error 34 us; skew 500 ppm
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:05.150036 13809 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 13:17:05.150565 13809 webserver.cc:460] Webserver started at http://127.0.98.61:39987/ using document root <none> and password file <none>
I20260926 13:17:05.150795 13809 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:05.150861 13809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:05.151021 13809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:05.152114 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data/instance:
uuid: "4578ad7471004f5dbcb3639fc4b4446d"
format_stamp: "Formatted at 2026-09-26 13:17:05 on dist-test-slave-f864"
server_key: "3943d93e3bb5d6ab2a3763a92fa44f14"
server_key_iv: "977121a104529ecaf385df48aca7a1c7"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.152486 13809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal/instance:
uuid: "4578ad7471004f5dbcb3639fc4b4446d"
format_stamp: "Formatted at 2026-09-26 13:17:05 on dist-test-slave-f864"
server_key: "3943d93e3bb5d6ab2a3763a92fa44f14"
server_key_iv: "977121a104529ecaf385df48aca7a1c7"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.153800 13809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:17:05.154578 13824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:05.154736 13809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:05.154831 13809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal
uuid: "4578ad7471004f5dbcb3639fc4b4446d"
format_stamp: "Formatted at 2026-09-26 13:17:05 on dist-test-slave-f864"
server_key: "3943d93e3bb5d6ab2a3763a92fa44f14"
server_key_iv: "977121a104529ecaf385df48aca7a1c7"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.154922 13809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:05.157511 13802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } attempt: 2
I20260926 13:17:05.157883 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.157765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34134)
0926 13:17:05.157818 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.157849 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:17:05.158192 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.158085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34142)
0926 13:17:05.158126 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.158148 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 13:17:05.158282 13802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42417: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42417: connect: Connection refused (error 111)
I20260926 13:17:05.176046 13809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:05.177930 13809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:05.178041 13809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:05.181854 13809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42417
I20260926 13:17:05.181916 13876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42417 every 8 connection(s)
I20260926 13:17:05.182283 13809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:17:05.183123 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:05.184646 13877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:17:05.184994 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13809
I20260926 13:17:05.185102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-1/wal/instance
I20260926 13:17:05.185276 392 external_mini_cluster.cc:1428] Setting key 1369f314119ffc81001d4983058e653e
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 13:17:05.194976 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.185401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43764 (local address 127.0.98.62:38071)
0926 13:17:05.185451 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.185453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.185759 (+ 306us) server_negotiation.cc:407] Connection header received
0926 13:17:05.186239 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.186241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.186257 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.186333 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.187163 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.188009 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.189522 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.189715 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.189767 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.192142 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.192144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.192144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.192152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.193594 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.193978 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.193979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.193979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.194046 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.194330 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.194331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.194331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.194454 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.194536 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.194803 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.194888 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:17:05.195147 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.185428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.0.1:43764)
0926 13:17:05.185595 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.185638 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.186109 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.186388 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.186391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.186762 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.187068 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.187075 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.188098 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.188099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.189438 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.189441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.189804 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.189805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.189976 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.190494 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.190505 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.192047 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.193683 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.193690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.193692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.193904 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.194121 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.194122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.194122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.194266 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.194527 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.194533 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.194673 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.194770 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.194921 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260926 13:17:05.195763 13877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:17:05.202993 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.196029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34154)
0926 13:17:05.196188 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.196196 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.196228 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.196389 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.196390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.196547 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.196586 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.196588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.197447 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.197448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.197919 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.197920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.198479 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.198480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.198550 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.198942 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.198947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.200275 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.201918 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.201918 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.201919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.202077 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.202315 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.202316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.202316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.202368 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.202684 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.202685 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.202722 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.202793 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.202887 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 13:17:05.203084 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.196061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.98.61:42417)
0926 13:17:05.196230 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.196234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.196260 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:17:05.196284 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.196286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.196301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.196389 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.196667 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.197355 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.198024 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.198392 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.198480 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.200367 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.200374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.200374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.200381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.201820 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.202185 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.202188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.202189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.202247 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.202469 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.202469 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.202470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.202606 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.202705 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.202823 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.202955 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260926 13:17:05.203662 13877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:17:05.204026 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.203893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55258)
0926 13:17:05.203944 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.203995 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:17:05.204200 13828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
I20260926 13:17:05.204691 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.204587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55264)
0926 13:17:05.204615 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.204670 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260926 13:17:05.204797 13877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 13:17:05.211400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45251
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38071,127.0.98.61:42417,127.0.98.60:45251
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:05.249447 13802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } attempt: 3
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:17:05.254029 13877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } attempt: 1
I20260926 13:17:05.254436 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.254329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55268)
0926 13:17:05.254377 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.254404 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:17:05.254763 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.254667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55272)
0926 13:17:05.254705 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.254740 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 13:17:05.254951 13877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
I20260926 13:17:05.256904 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.249858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34158)
0926 13:17:05.249923 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.249934 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.250006 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.250154 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.250155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.250337 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.250399 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.250401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.251095 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.251097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.251609 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.251611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.251990 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.251992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.252077 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.252576 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.252581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.253990 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.255993 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.255994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.255994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.256159 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.256362 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.256363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.256363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.256413 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.256670 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.256671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.256712 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.256763 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.256838 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:17:05.256963 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.249935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.0.98.61:42417)
0926 13:17:05.250004 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.250007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.250018 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:05.250071 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.250072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.250084 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.250151 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.250506 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.251007 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.251699 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.251908 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.251985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.254084 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.254085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.254086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.254098 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.255887 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.256228 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.256229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.256230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.256296 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.256471 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.256472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.256473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.256609 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.256675 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.256806 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.256884 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 13:17:05.257701 13802 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:17:05.258059 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.257941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55284)
0926 13:17:05.257991 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.258032 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:17:05.258416 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.258318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55296)
0926 13:17:05.258365 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.258396 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:17:05.258536 13802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
I20260926 13:17:05.289717 13802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } attempt: 1
I20260926 13:17:05.290239 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.290053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55312)
0926 13:17:05.290121 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.290198 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 13:17:05.290598 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.290502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55316)
0926 13:17:05.290544 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.290567 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 13:17:05.290690 13802 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
W20260926 13:17:05.294219 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:05.294418 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:05.294461 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:05.295625 13884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:05.295665 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:05.295679 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:05.295715 13884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:05.295746 13884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:05.297238 13884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38071,127.0.98.61:42417,127.0.98.60:45251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:05.297483 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:05.297729 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:05.302559 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:05.302546 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:05.302533 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:05.302881 13884 server_base.cc:1034] running on GCE node
I20260926 13:17:05.303136 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:05.303423 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:05.304585 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790428625304572 us; error 27 us; skew 500 ppm
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:05.307211 13884 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 13:17:05.307592 13884 webserver.cc:460] Webserver started at http://127.0.98.60:39427/ using document root <none> and password file <none>
I20260926 13:17:05.307780 13884 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:05.307819 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:05.307906 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:05.308919 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data/instance:
uuid: "8088ddcfd26344f8b8a661ffb12edc84"
format_stamp: "Formatted at 2026-09-26 13:17:05 on dist-test-slave-f864"
server_key: "0be86cc1b50ed9725172d8ddc5653fba"
server_key_iv: "87e18edceade53048cdd697e1136cde3"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.309226 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal/instance:
uuid: "8088ddcfd26344f8b8a661ffb12edc84"
format_stamp: "Formatted at 2026-09-26 13:17:05 on dist-test-slave-f864"
server_key: "0be86cc1b50ed9725172d8ddc5653fba"
server_key_iv: "87e18edceade53048cdd697e1136cde3"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.310483 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:17:05.311272 13899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:05.311477 13884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:05.311579 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal
uuid: "8088ddcfd26344f8b8a661ffb12edc84"
format_stamp: "Formatted at 2026-09-26 13:17:05 on dist-test-slave-f864"
server_key: "0be86cc1b50ed9725172d8ddc5653fba"
server_key_iv: "87e18edceade53048cdd697e1136cde3"
server_key_version: "encryptionkey@0"
I20260926 13:17:05.311653 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:05.323107 13877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } attempt: 2
I20260926 13:17:05.323537 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.323407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55318)
0926 13:17:05.323468 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.323507 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:17:05.323869 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.323775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55324)
0926 13:17:05.323820 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.323844 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:17:05.323959 13877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45251: connect: Connection refused (error 111)
I20260926 13:17:05.340934 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:05.343062 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:05.343225 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:05.347270 13884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45251
I20260926 13:17:05.347304 13951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45251 every 8 connection(s)
I20260926 13:17:05.347662 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:17:05.348553 13952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:05.350088 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:05.356568 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 13884
I20260926 13:17:05.356691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790428444292953-392-0/minicluster-data/master-2/wal/instance
I20260926 13:17:05.356912 392 external_mini_cluster.cc:1428] Setting key 21c246eb9f24f3587b58f2f7ef4f1590
I20260926 13:17:05.359407 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.350873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.98.62:38071)
0926 13:17:05.350922 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.350924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.351159 (+ 235us) server_negotiation.cc:407] Connection header received
0926 13:17:05.351679 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.351680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.351693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.351739 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.352365 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.352835 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.353811 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.354006 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.354084 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.356186 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.356187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.356187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.356194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.357913 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.358452 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.358453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.358454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.358523 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.358761 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.358762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.358762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.358902 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.358972 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.359226 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.359312 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:17:05.359687 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.350861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38071 (local address 127.0.0.1:43776)
0926 13:17:05.351034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.351071 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.351559 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.351822 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.351826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.352078 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.352270 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.352274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.352928 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.352929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.353718 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.353720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.354089 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.354090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.354275 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.354725 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.354733 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.356097 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.357983 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.357990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.357993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.358373 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.358595 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.358595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.358596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.358694 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.358967 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.358971 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.359090 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.359175 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.359408 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":101,"thread_start_us":80,"threads_started":1}
I20260926 13:17:05.360306 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:17:05.365832 13802 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } attempt: 2
I20260926 13:17:05.366758 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.357296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.0.98.60:45251)
0926 13:17:05.357479 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.357484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.357552 (+ 68us) server_negotiation.cc:407] Connection header received
0926 13:17:05.357583 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.357586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.357601 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.357677 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.358453 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.359515 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.360444 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.360951 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.361035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.363596 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.363602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.363602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.363608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.365388 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.365786 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.365788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.365789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.365840 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.366087 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.366088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.366088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.366309 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.366479 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.366550 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.366625 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260926 13:17:05.367969 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.360640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34168 (local address 127.0.98.61:42417)
0926 13:17:05.360679 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.360681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.360689 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:17:05.360727 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.360728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.360738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.360793 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.361023 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.361510 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.362526 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.362696 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.362783 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.364527 (+ 1744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.364528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.364529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.364536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.366119 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.366576 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.366577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.366578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.366633 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.367437 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.367438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.367438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.367567 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.367612 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.367806 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.367882 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:17:05.368254 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.360572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42417 (local address 127.0.0.1:34168)
0926 13:17:05.360620 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.360630 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.360664 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.360794 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.360794 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.360899 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.360944 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.360945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.361568 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.361569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.362430 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.362433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.362783 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.362784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.362856 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.363274 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.363280 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.364433 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.366281 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.366282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.366282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.366493 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.366704 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.366704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.366705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.366756 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.367662 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.367663 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.367701 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.368059 (+ 358us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.368143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:17:05.368638 13952 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:05.373739 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.366082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55332)
0926 13:17:05.366120 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.366132 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.366188 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.366499 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.366500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.366669 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.366719 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.366722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.367376 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.367378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.368167 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.368170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.368669 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.368671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.368729 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.369131 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.369136 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.370367 (+ 1231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.372639 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.372640 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.372640 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.372858 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.373091 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.373092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.373092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.373142 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.373407 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.373408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.373450 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.373537 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.373627 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 13:17:05.373765 13957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.366195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.0.98.60:45251)
0926 13:17:05.366364 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.366366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.366373 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:17:05.366390 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.366391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.366401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.366793 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.366822 (+ 29us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.367298 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.368255 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.368575 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.368662 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.370449 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.370450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.370451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.370457 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.372559 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.372945 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.372946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.372947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.373019 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.373202 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.373203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.373203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.373339 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.373399 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.373566 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.373641 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
I20260926 13:17:05.376134 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.368873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55344)
0926 13:17:05.368907 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.368917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.368953 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.369046 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.369046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.369187 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.369212 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.369214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.370090 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.370090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.370783 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.370786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.371295 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.371296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.371353 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.371701 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.371705 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.373584 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.375175 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.375175 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.375175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.375328 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.375524 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.375525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.375526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.375581 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.375836 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.375837 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.375872 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.375938 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.376030 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 13:17:05.376189 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.368936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.0.98.60:45251)
0926 13:17:05.368962 (+ 26us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.368963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.368973 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:05.368995 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.368996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.369004 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.369034 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.369309 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.370010 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.370870 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.371197 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.371261 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.373663 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.373664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.373665 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.373670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.375102 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.375398 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.375399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.375399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.375448 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.375654 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.375655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.375655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.375771 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.375843 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.375972 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.376074 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":12}
I20260926 13:17:05.376706 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.367721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.98.62:38071)
0926 13:17:05.367756 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.367758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.367766 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:17:05.367823 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.367824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.367832 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.367885 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.368535 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.369193 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.370192 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.370481 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.370565 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.373797 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.373799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.373799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.373805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.375497 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.375858 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.375859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.375859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.375907 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.376108 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.376108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.376109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.376239 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.376397 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.376596 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.376653 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:17:05.376848 13802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: Bootstrap starting.
I20260926 13:17:05.377483 13802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:05.377787 13802 log.cc:824] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:05.379110 13802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: No bootstrap required, opened a new log
I20260926 13:17:05.379374 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: Bootstrap starting.
I20260926 13:17:05.380048 13952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:05.380373 13952 log.cc:824] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:05.381347 13952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: No bootstrap required, opened a new log
I20260926 13:17:05.381418 13802 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.381562 13802 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:05.381596 13802 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3d481e2c02439181124b55b6636bfd, State: Initialized, Role: FOLLOWER
I20260926 13:17:05.381731 13802 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:17:05.382057 13802 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:05.382036 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.382301 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:17:05.383639 13952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.383800 13952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:05.383841 13952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8088ddcfd26344f8b8a661ffb12edc84, State: Initialized, Role: FOLLOWER
I20260926 13:17:05.383971 13952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.384212 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.384261 13952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:05.384275 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:17:05.385279 13971 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:17:05.385557 13971 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:17:05.385890 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.377248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.0.98.61:42417)
0926 13:17:05.377293 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.377294 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.377353 (+ 59us) server_negotiation.cc:407] Connection header received
0926 13:17:05.377422 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.377423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.377431 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.377471 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.378027 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.378663 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.379379 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.379650 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.379726 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.382482 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.382483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.382484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.382491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.384252 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.384678 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.384679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.384679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.384738 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.384946 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.384947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.384947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.385081 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.385226 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.385701 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.385806 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 13:17:05.386876 13984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:17:05.386972 13984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:17:05.385155 13734 master_runner.cc:186] Error getting master registration for 127.0.98.61:42417: OK, instance_id { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" instance_seqno: 1790428625180785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:17:05.388204 13734 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:17:05.389351 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:17:05.390841 13884 master_runner.cc:186] Error getting master registration for 127.0.98.61:42417: OK, instance_id { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" instance_seqno: 1790428625180785 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:17:05.393114 13884 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:17:05.399616 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.390854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.0.98.60:45251)
0926 13:17:05.390894 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.390895 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.391475 (+ 580us) server_negotiation.cc:407] Connection header received
0926 13:17:05.391570 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.391571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.391581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.391611 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.392378 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.393121 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.394180 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.394465 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.394537 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.396112 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.396114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.396114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.396123 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.398362 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.398731 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.398733 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.398733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.398800 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.398992 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.398992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.398992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.399113 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.399208 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.399440 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.399521 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:17:05.401724 13881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.391205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34200 (local address 127.0.98.61:42417)
0926 13:17:05.391241 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.391243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.394647 (+ 3404us) server_negotiation.cc:407] Connection header received
0926 13:17:05.394742 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.394743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.394753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.394783 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.395492 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.396201 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.397216 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.397510 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.397608 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.398898 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.398900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.398900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.398906 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.400527 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.400925 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.400927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.400927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.400982 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.401173 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.401174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.401174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.401294 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.401362 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.401552 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.401628 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:17:05.407826 13806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.394356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43800 (local address 127.0.98.62:38071)
0926 13:17:05.394407 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.394409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.397344 (+ 2935us) server_negotiation.cc:407] Connection header received
0926 13:17:05.397453 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.397454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.397467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.397501 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.401098 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.401835 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.403199 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.403503 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.403578 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.404947 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.404948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.404949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.404956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.406660 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.407018 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.407018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.407019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.407078 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.407278 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.407278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.407279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.407409 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.407515 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.407651 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.407723 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:17:05.415056 13877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } attempt: 3
Sep 26 13:17:05 dist-test-slave-f864 krb5kdc[13721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:17:05.423216 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.415379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45251 (local address 127.0.0.1:55360)
0926 13:17:05.415428 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:05.415438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:05.415474 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:05.415658 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:05.415659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:05.415847 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:05.415894 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.415897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.416804 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.416805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.417605 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:05.417607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.418074 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.418075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:05.418158 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.418709 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:05.418714 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:05.420206 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:05.422049 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.422050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.422050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.422291 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.422518 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:05.422519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:05.422519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:05.422585 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:05.422894 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:05.422895 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:05.422943 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 13:17:05.423018 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:17:05.423112 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 13:17:05.423233 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:05.415433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.98.60:45251)
0926 13:17:05.415474 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:05.415476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:05.415486 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:05.415539 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:05.415539 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:05.415549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:05.415608 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:05.415979 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.416715 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.417691 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:05.417991 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:05.418064 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:05.420300 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:05.420301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:05.420301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:05.420309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:05.421958 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.422384 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.422385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.422386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.422447 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:05.422653 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:05.422654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:05.422654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:05.422816 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:05.422885 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:05.423057 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 13:17:05.423135 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:17:05.425122 13877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: Bootstrap starting.
I20260926 13:17:05.425894 13877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:05.426271 13877 log.cc:824] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:05.427570 13877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: No bootstrap required, opened a new log
I20260926 13:17:05.429567 13877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.429716 13877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:05.429751 13877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4578ad7471004f5dbcb3639fc4b4446d, State: Initialized, Role: FOLLOWER
I20260926 13:17:05.429888 13877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.430235 13877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:05.430153 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.430393 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:17:05.432526 14005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:17:05.432624 14005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:17:05.434525 13809 master_runner.cc:263] No leader master found from master 4578ad7471004f5dbcb3639fc4b4446d
I20260926 13:17:05.434579 13809 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 13:17:05.518599 13755 tablet.cc:2369] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:17:05.593880 13905 tablet.cc:2369] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:17:05.598371 13960 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:17:05.598520 13960 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.598816 13960 leader_election.cc:288] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4578ad7471004f5dbcb3639fc4b4446d (127.0.98.61:42417), 8088ddcfd26344f8b8a661ffb12edc84 (127.0.98.60:45251)
I20260926 13:17:05.599166 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac3d481e2c02439181124b55b6636bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8088ddcfd26344f8b8a661ffb12edc84" is_pre_election: true
I20260926 13:17:05.599206 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac3d481e2c02439181124b55b6636bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4578ad7471004f5dbcb3639fc4b4446d" is_pre_election: true
I20260926 13:17:05.599354 13851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d481e2c02439181124b55b6636bfd in term 0.
I20260926 13:17:05.599365 13926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3d481e2c02439181124b55b6636bfd in term 0.
I20260926 13:17:05.599584 13753 leader_election.cc:302] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8088ddcfd26344f8b8a661ffb12edc84, ac3d481e2c02439181124b55b6636bfd; no voters:
I20260926 13:17:05.599769 13960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:17:05.599871 13960 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:17:05.599907 13960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:05.600665 13960 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.600832 13960 leader_election.cc:288] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4578ad7471004f5dbcb3639fc4b4446d (127.0.98.61:42417), 8088ddcfd26344f8b8a661ffb12edc84 (127.0.98.60:45251)
I20260926 13:17:05.601001 13926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac3d481e2c02439181124b55b6636bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8088ddcfd26344f8b8a661ffb12edc84"
I20260926 13:17:05.601079 13926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:05.601254 13851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ac3d481e2c02439181124b55b6636bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4578ad7471004f5dbcb3639fc4b4446d"
I20260926 13:17:05.601341 13851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:05.601946 13926 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d481e2c02439181124b55b6636bfd in term 1.
I20260926 13:17:05.602123 13753 leader_election.cc:302] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8088ddcfd26344f8b8a661ffb12edc84, ac3d481e2c02439181124b55b6636bfd; no voters:
I20260926 13:17:05.602259 13960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:05.602284 13851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3d481e2c02439181124b55b6636bfd in term 1.
I20260926 13:17:05.602394 13960 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [term 1 LEADER]: Becoming Leader. State: Replica: ac3d481e2c02439181124b55b6636bfd, State: Running, Role: LEADER
I20260926 13:17:05.602488 13960 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } }
I20260926 13:17:05.602967 14007 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3d481e2c02439181124b55b6636bfd. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.603101 14007 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:05.603343 14009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:05.603534 14009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:05.604900 13851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [term 1 FOLLOWER]: Refusing update from remote peer ac3d481e2c02439181124b55b6636bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:05.604918 13926 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [term 1 FOLLOWER]: Refusing update from remote peer ac3d481e2c02439181124b55b6636bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:05.605206 14007 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:05.605304 13960 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:05.606493 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3d481e2c02439181124b55b6636bfd. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.606606 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:17:05.606564 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3d481e2c02439181124b55b6636bfd. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.606652 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:17:05.607537 14007 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.607625 14007 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:05.607761 13994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.607841 13994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:17:05.607950 13960 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.608036 13960 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:05.608155 14010 mvcc.cc:205] Tried to move back new op lower bound from 7333595650475601920 to 7333595650468499456. Current Snapshot: MvccSnapshot[applied={T|T < 7333595650475601920}]
I20260926 13:17:05.608237 14012 mvcc.cc:205] Tried to move back new op lower bound from 7333595650475601920 to 7333595650468499456. Current Snapshot: MvccSnapshot[applied={T|T < 7333595650475601920 or (T in {7333595650475601920})}]
I20260926 13:17:05.608322 13972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ac3d481e2c02439181124b55b6636bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac3d481e2c02439181124b55b6636bfd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38071 } } peers { permanent_uuid: "4578ad7471004f5dbcb3639fc4b4446d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42417 } } peers { permanent_uuid: "8088ddcfd26344f8b8a661ffb12edc84" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45251 } } }
I20260926 13:17:05.608407 13972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:17:05.608647 14011 mvcc.cc:205] Tried to move back new op lower bound from 7333595650475601920 to 7333595650468499456. Current Snapshot: MvccSnapshot[applied={T|T < 7333595650475601920 or (T in {7333595650475601920})}]
I20260926 13:17:05.610581 14009 catalog_manager.cc:1347] Generated new cluster ID: b25bfae847e14b4d83dbb244eea017d9
I20260926 13:17:05.610641 14009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:17:05.625725 14009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:17:05.626380 14009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:05.629484 14009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3d481e2c02439181124b55b6636bfd: Generated new TSK 0
I20260926 13:17:05.629678 14009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:17:06.387598 13984 catalog_manager.cc:1259] Loaded cluster ID: b25bfae847e14b4d83dbb244eea017d9
I20260926 13:17:06.387702 13984 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: loading cluster ID for follower catalog manager: success
I20260926 13:17:06.389113 13984 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: acquiring CA information for follower catalog manager: success
I20260926 13:17:06.389492 13984 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8088ddcfd26344f8b8a661ffb12edc84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:17:06.433102 14005 catalog_manager.cc:1259] Loaded cluster ID: b25bfae847e14b4d83dbb244eea017d9
I20260926 13:17:06.433172 14005 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: loading cluster ID for follower catalog manager: success
I20260926 13:17:06.433913 14005 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: acquiring CA information for follower catalog manager: success
I20260926 13:17:06.434198 14005 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4578ad7471004f5dbcb3639fc4b4446d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:17:06.687353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13734
I20260926 13:17:06.692534 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13809
I20260926 13:17:06.697590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 13884
2026-09-26T13:17:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2974 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:17:06 dist-test-slave-f864 krb5kdc[14024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:17:06 dist-test-slave-f864 krb5kdc[14024](info): set up 2 sockets
Sep 26 13:17:06 dist-test-slave-f864 krb5kdc[14024](info): commencing operation
krb5kdc: starting...
W20260926 13:17:07.734302 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:17:07 dist-test-slave-f864 krb5kdc[14024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:17:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:17:07.854015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:07.934906 14037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:07.935111 14037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:07.935133 14037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:07.936280 14037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:07.936321 14037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:07.936335 14037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:07.936348 14037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:07.936360 14037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:07.937803 14037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:07.938074 14037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:07.938367 14037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:07.943130 14043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:07.943272 14037 server_base.cc:1034] running on GCE node
W20260926 13:17:07.943188 14042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:07.943127 14045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:07.943591 14037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:07.943866 14037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:07.945024 14037 hybrid_clock.cc:648] HybridClock initialized: now 1790428627944988 us; error 36 us; skew 500 ppm
Sep 26 13:17:07 dist-test-slave-f864 krb5kdc[14024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:07.947417 14037 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:07.947826 14037 webserver.cc:460] Webserver started at http://127.0.98.62:38815/ using document root <none> and password file <none>
I20260926 13:17:07.948051 14037 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:07.948120 14037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:07.948267 14037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:07.949287 14037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "0354996fee0e4ba18cd81ee8d16b32a3"
format_stamp: "Formatted at 2026-09-26 13:17:07 on dist-test-slave-f864"
server_key: "c813f9c1e11c2eba882c7d18a7f40af8"
server_key_iv: "ea538dc8a72fff239ff48ac02c199d83"
server_key_version: "encryptionkey@0"
I20260926 13:17:07.949642 14037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "0354996fee0e4ba18cd81ee8d16b32a3"
format_stamp: "Formatted at 2026-09-26 13:17:07 on dist-test-slave-f864"
server_key: "c813f9c1e11c2eba882c7d18a7f40af8"
server_key_iv: "ea538dc8a72fff239ff48ac02c199d83"
server_key_version: "encryptionkey@0"
I20260926 13:17:07.951010 14037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:17:07.951766 14052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:07.951962 14037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:07.952064 14037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "0354996fee0e4ba18cd81ee8d16b32a3"
format_stamp: "Formatted at 2026-09-26 13:17:07 on dist-test-slave-f864"
server_key: "c813f9c1e11c2eba882c7d18a7f40af8"
server_key_iv: "ea538dc8a72fff239ff48ac02c199d83"
server_key_version: "encryptionkey@0"
I20260926 13:17:07.952138 14037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:07.978386 14037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:07.980502 14037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:07.980687 14037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:07.984856 14037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45805
I20260926 13:17:07.984894 14104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45805 every 8 connection(s)
I20260926 13:17:07.985253 14037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:07.986217 14105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:07.988510 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3: Bootstrap starting.
I20260926 13:17:07.989092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14037
I20260926 13:17:07.989125 14105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:07.989185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:17:07.989394 392 external_mini_cluster.cc:1428] Setting key e239d3ebcb360490a20657328dde20d2
I20260926 13:17:07.989449 14105 log.cc:824] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:07.990417 14105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3: No bootstrap required, opened a new log
I20260926 13:17:07.992897 14105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260926 13:17:07.993072 14105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:07.993114 14105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0354996fee0e4ba18cd81ee8d16b32a3, State: Initialized, Role: FOLLOWER
I20260926 13:17:07.993312 14105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260926 13:17:07.993400 14105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:07.993423 14105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:07.993498 14105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:07.994097 14105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260926 13:17:07.994226 14105 leader_election.cc:302] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0354996fee0e4ba18cd81ee8d16b32a3; no voters:
I20260926 13:17:07.994391 14105 leader_election.cc:288] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:17:07.994452 14110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:07.994648 14105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:07.994632 14110 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [term 1 LEADER]: Becoming Leader. State: Replica: 0354996fee0e4ba18cd81ee8d16b32a3, State: Running, Role: LEADER
I20260926 13:17:07.994869 14110 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } }
I20260926 13:17:07.995227 14111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } } }
I20260926 13:17:07.995342 14111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:07.995373 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0354996fee0e4ba18cd81ee8d16b32a3. Latest consensus state: current_term: 1 leader_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0354996fee0e4ba18cd81ee8d16b32a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45805 } } }
I20260926 13:17:07.995451 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:07.996142 14119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:07.996785 14119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 13:17:07 dist-test-slave-f864 krb5kdc[14024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:08.000044 14119 catalog_manager.cc:1347] Generated new cluster ID: 7eafb0608ddc4a979d93a8ee78e58c33
I20260926 13:17:08.000098 14119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:17:08.001484 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:07.989927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.0.98.62:45805)
0926 13:17:07.990091 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:07.990100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:07.990134 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:17:07.990795 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:07.990805 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:07.990867 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:07.991045 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:07.991608 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:07.993068 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:07.993937 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:07.994602 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:07.995069 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:07.997802 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:07.997806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:07.997806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:07.997818 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:07.999733 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:08.000155 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:08.000158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:08.000159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:08.000256 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:08.000556 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:08.000557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:08.000558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:08.000840 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:08.000980 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:08.001141 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 13:17:08.001271 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260926 13:17:08.002506 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:17:08.009068 14119 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:17:08.009562 14119 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:08.019731 14119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0354996fee0e4ba18cd81ee8d16b32a3: Generated new TSK 0
I20260926 13:17:08.019958 14119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:17:08.056898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14037
2026-09-26T13:17:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1357 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:17:08 dist-test-slave-f864 krb5kdc[14132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:17:08 dist-test-slave-f864 krb5kdc[14132](info): set up 2 sockets
Sep 26 13:17:08 dist-test-slave-f864 krb5kdc[14132](info): commencing operation
krb5kdc: starting...
W20260926 13:17:09.086336 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:17:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:17:09.200644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:09.281622 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:09.281860 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:09.281898 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:09.283102 14145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:09.283178 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:09.283202 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:09.283241 14145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:09.283293 14145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:09.284746 14145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:09.285027 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:09.285267 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:17:09.290208 14145 server_base.cc:1034] running on GCE node
W20260926 13:17:09.290141 14150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:09.290271 14151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:09.290149 14153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:09.290611 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:09.290901 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:09.292074 14145 hybrid_clock.cc:648] HybridClock initialized: now 1790428629292079 us; error 43 us; skew 500 ppm
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:09.294773 14145 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:09.295229 14145 webserver.cc:460] Webserver started at http://127.0.98.62:33691/ using document root <none> and password file <none>
I20260926 13:17:09.295501 14145 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:09.295572 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:09.295724 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:09.297331 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "e686222b064244fca0b4fa1cc7b253c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "fdc0adb125f802f04dcd201de58be578"
server_key_iv: "bb50ffefc37a04e9ce17beb0f79caea3"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.297719 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "e686222b064244fca0b4fa1cc7b253c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "fdc0adb125f802f04dcd201de58be578"
server_key_iv: "bb50ffefc37a04e9ce17beb0f79caea3"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.299154 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 13:17:09.299965 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.300172 14145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:17:09.300275 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "e686222b064244fca0b4fa1cc7b253c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "fdc0adb125f802f04dcd201de58be578"
server_key_iv: "bb50ffefc37a04e9ce17beb0f79caea3"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.300359 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:09.317349 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:09.319510 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:09.319677 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:09.323828 14145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43329
I20260926 13:17:09.323858 14212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43329 every 8 connection(s)
I20260926 13:17:09.324249 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:09.325106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14145
I20260926 13:17:09.325232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:17:09.325300 14213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:09.325446 392 external_mini_cluster.cc:1428] Setting key d7ea879b0fd228da67e70a37cfa1cf52
I20260926 13:17:09.328689 14213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap starting.
I20260926 13:17:09.329438 14213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:09.329751 14213 log.cc:824] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:09.330529 14213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: No bootstrap required, opened a new log
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:09.333129 14213 raft_consensus.cc:348] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:09.333290 14213 raft_consensus.cc:374] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:09.333366 14213 raft_consensus.cc:729] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e686222b064244fca0b4fa1cc7b253c9, State: Initialized, Role: FOLLOWER
I20260926 13:17:09.333547 14213 consensus_queue.cc:260] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:09.333648 14213 raft_consensus.cc:388] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:09.333690 14213 raft_consensus.cc:482] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:09.333741 14213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:09.334344 14213 raft_consensus.cc:504] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:09.334498 14213 leader_election.cc:302] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e686222b064244fca0b4fa1cc7b253c9; no voters:
I20260926 13:17:09.334690 14213 leader_election.cc:288] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:17:09.334722 14218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:09.334906 14218 raft_consensus.cc:686] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 LEADER]: Becoming Leader. State: Replica: e686222b064244fca0b4fa1cc7b253c9, State: Running, Role: LEADER
I20260926 13:17:09.334976 14213 sys_catalog.cc:564] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:09.335029 14218 consensus_queue.cc:237] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:09.335350 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e686222b064244fca0b4fa1cc7b253c9. Latest consensus state: current_term: 1 leader_uuid: "e686222b064244fca0b4fa1cc7b253c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260926 13:17:09.335439 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:09.335361 14219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e686222b064244fca0b4fa1cc7b253c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260926 13:17:09.335626 14219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:09.335819 14225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:09.336526 14225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:09.337939 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.326020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.0.98.62:43329)
0926 13:17:09.326220 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.326233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.326260 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:17:09.326788 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.326795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.326846 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.326968 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:09.327754 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.329135 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.330122 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.330654 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.330760 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.333375 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:09.333380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:09.333380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:09.333390 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:09.335566 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.336164 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.336168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.336169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.336298 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.336529 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.336530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.336530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.337018 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:09.337155 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.337424 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.337800 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260926 13:17:09.338275 14225 catalog_manager.cc:1347] Generated new cluster ID: 2521171d6a304750b7bc1ef7d2559840
I20260926 13:17:09.338325 14225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:17:09.350741 14225 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:17:09.351348 14225 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:09.354535 14225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Generated new TSK 0
I20260926 13:17:09.354728 14225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:17:09.384274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:09.464633 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:09.464833 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:09.464854 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:09.465919 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:09.465952 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:09.465978 14239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:17:09.467341 14239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:09.467531 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:09.467713 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:09.472769 14244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:09.472812 14245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:09.472882 14239 server_base.cc:1034] running on GCE node
W20260926 13:17:09.472766 14247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:09.473210 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:09.473490 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:09.474650 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790428629474633 us; error 46 us; skew 500 ppm
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:09.477028 14239 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:17:09.477475 14239 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260926 13:17:09.477710 14239 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:09.477779 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:09.477933 14239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:09.479053 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "69313ddfd226493da7f9ee1628837030"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "4f8cc71cd797c48d569a3ff56a9e4071"
server_key_iv: "ccdd52beab58f6900f983c6fb04ee6f2"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.479414 14239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "69313ddfd226493da7f9ee1628837030"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "4f8cc71cd797c48d569a3ff56a9e4071"
server_key_iv: "ccdd52beab58f6900f983c6fb04ee6f2"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.480690 14239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:17:09.481393 14254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.481575 14239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:17:09.481673 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "69313ddfd226493da7f9ee1628837030"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "4f8cc71cd797c48d569a3ff56a9e4071"
server_key_iv: "ccdd52beab58f6900f983c6fb04ee6f2"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.481758 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:09.507282 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:09.509193 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:09.509364 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:09.509614 14239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:09.509935 14239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:09.509994 14239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.510033 14239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:09.510061 14239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.515708 14239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42297
I20260926 13:17:09.515770 14367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42297 every 8 connection(s)
I20260926 13:17:09.516078 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:17:09.519146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14239
I20260926 13:17:09.519280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:17:09.519486 392 external_mini_cluster.cc:1428] Setting key 65a6ed36fdbdeea77cb015df40b46a5b
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:09.526350 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.516809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43423 (local address 127.0.98.62:43329)
0926 13:17:09.516870 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.516872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.517080 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:17:09.517553 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.517554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.517563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.517621 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:09.518563 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.519076 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.520411 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.520625 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.520700 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.523262 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:09.523264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:09.523264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:09.523273 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:09.524730 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.525266 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.525267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.525268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.525319 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.525623 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.525624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.525624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.525723 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:09.525773 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.526141 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.526286 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:17:09.526585 14370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.516778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:43423)
0926 13:17:09.516943 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:09.516984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:09.517469 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:09.517674 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:09.517678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:09.518280 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:09.518482 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.518487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.519185 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.519187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.520317 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.520321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.520708 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.520709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.520836 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.521432 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:09.521445 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:09.523155 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:09.524818 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:09.524827 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:09.524830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:09.525183 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:09.525389 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:09.525390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:09.525390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:09.525551 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:09.525789 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:09.525796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:09.526000 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 13:17:09.526118 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 13:17:09.526325 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260926 13:17:09.527153 14368 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260926 13:17:09.527364 14368 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:09.527552 14368 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:17:09.528167 14177 ts_manager.cc:194] Registered new tserver with Master: 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297)
I20260926 13:17:09.528898 14177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:17:09.545814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:09.626734 14373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:09.627002 14373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:09.627043 14373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:09.628228 14373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:09.628384 14373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:09.628448 14373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:09.629868 14373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:09.630052 14373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:09.630307 14373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:09.635226 14378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:09.635360 14373 server_base.cc:1034] running on GCE node
W20260926 13:17:09.635233 14381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:09.635216 14379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:09.635773 14373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:09.636138 14373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:09.637310 14373 hybrid_clock.cc:648] HybridClock initialized: now 1790428629637300 us; error 41 us; skew 500 ppm
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:09.639820 14373 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:09.640339 14373 webserver.cc:460] Webserver started at http://127.0.98.2:39409/ using document root <none> and password file <none>
I20260926 13:17:09.640570 14373 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:09.640640 14373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:09.640797 14373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:09.641856 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "485f154f73dc4a12bdf2032e94cee6c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "3d3eed8a7586df6aa0bcf7d9c12e5c5a"
server_key_iv: "6fcab3809b89975577f6dc4308f13c18"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.642311 14373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "485f154f73dc4a12bdf2032e94cee6c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "3d3eed8a7586df6aa0bcf7d9c12e5c5a"
server_key_iv: "6fcab3809b89975577f6dc4308f13c18"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.643618 14373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:09.644361 14388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.644553 14373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:17:09.644650 14373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "485f154f73dc4a12bdf2032e94cee6c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "3d3eed8a7586df6aa0bcf7d9c12e5c5a"
server_key_iv: "6fcab3809b89975577f6dc4308f13c18"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.644738 14373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:09.661417 14373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:09.663655 14373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:09.663882 14373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:09.664173 14373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:09.664565 14373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:09.664628 14373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.664674 14373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:09.664709 14373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.671298 14373 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38121
I20260926 13:17:09.671337 14501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38121 every 8 connection(s)
I20260926 13:17:09.671705 14373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:09.680925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14373
I20260926 13:17:09.681039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:17:09.681154 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.672620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57847 (local address 127.0.98.62:43329)
0926 13:17:09.672685 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.672688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.672897 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:17:09.673429 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.673430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.673441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.673504 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:09.674514 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.675004 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.676094 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.676273 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.676323 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.678545 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:09.678547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:09.678547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:09.678554 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:09.679829 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.680221 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.680222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.680222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.680274 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.680517 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.680517 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.680518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.680622 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:09.680673 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.680961 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.681052 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:17:09.681260 392 external_mini_cluster.cc:1428] Setting key 1714c7a05facf5408a96ddf3eb047670
I20260926 13:17:09.681391 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.672567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:57847)
0926 13:17:09.672752 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:09.672799 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:09.673316 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:09.673565 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:09.673568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:09.674208 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:09.674433 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.674438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.675101 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.675102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.676021 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.676023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.676347 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.676348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.676473 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.676935 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:09.676944 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:09.678461 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:09.679910 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:09.679916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:09.679918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:09.680148 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:09.680338 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:09.680339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:09.680339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:09.680442 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:09.680683 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:09.680688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:09.680814 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 13:17:09.680906 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 13:17:09.681082 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260926 13:17:09.682116 14502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260926 13:17:09.682235 14502 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:09.682446 14502 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260926 13:17:09.682830 14177 ts_manager.cc:194] Registered new tserver with Master: 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2:38121)
I20260926 13:17:09.683259 14177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57847
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:17:09.704427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:09.785509 14507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:09.785712 14507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:09.785734 14507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:09.786839 14507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:09.786875 14507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:09.786899 14507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:09.788249 14507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:09.788491 14507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:09.788677 14507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:09.793537 14515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:09.793565 14512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:09.793543 14513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:09.793664 14507 server_base.cc:1034] running on GCE node
I20260926 13:17:09.793999 14507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:09.794299 14507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:09.795452 14507 hybrid_clock.cc:648] HybridClock initialized: now 1790428629795435 us; error 33 us; skew 500 ppm
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:09.797955 14507 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:09.798399 14507 webserver.cc:460] Webserver started at http://127.0.98.3:36401/ using document root <none> and password file <none>
I20260926 13:17:09.798645 14507 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:09.798689 14507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:09.798775 14507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:09.799741 14507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "433883ff49c34258b42293dcc698fde3"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "84b7f60923555d0f3c684d2ff910c462"
server_key_iv: "a847124a38b791e017b0265d1f98d175"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.800122 14507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "433883ff49c34258b42293dcc698fde3"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "84b7f60923555d0f3c684d2ff910c462"
server_key_iv: "a847124a38b791e017b0265d1f98d175"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.801321 14507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:17:09.801972 14522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.802143 14507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:17:09.802241 14507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "433883ff49c34258b42293dcc698fde3"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "84b7f60923555d0f3c684d2ff910c462"
server_key_iv: "a847124a38b791e017b0265d1f98d175"
server_key_version: "encryptionkey@0"
I20260926 13:17:09.802294 14507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:09.814868 14507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:09.816722 14507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:09.816838 14507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:09.817035 14507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:09.817304 14507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:09.817334 14507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.817358 14507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:09.817370 14507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:09.823269 14507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32983
I20260926 13:17:09.823345 14635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32983 every 8 connection(s)
I20260926 13:17:09.823707 14507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:17:09.829048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14507
I20260926 13:17:09.829155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:17:09.829375 392 external_mini_cluster.cc:1428] Setting key ae9ddc23097f772516426705d33aee48
Sep 26 13:17:09 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428629, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:09.833098 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.824512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50211 (local address 127.0.98.62:43329)
0926 13:17:09.824558 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.824560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.824724 (+ 164us) server_negotiation.cc:407] Connection header received
0926 13:17:09.825189 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.825190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.825199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.825256 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:09.826134 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.826687 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.827630 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.827803 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.827853 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.830474 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:09.830474 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:09.830475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:09.830481 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:09.831641 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.832059 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.832060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.832060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.832118 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.832390 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.832391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.832392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.832518 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:09.832590 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.832922 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.833004 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:17:09.833299 14638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.824427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:50211)
0926 13:17:09.824597 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:09.824639 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:09.825108 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:09.825309 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:09.825312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:09.825874 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:09.826059 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.826064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.826769 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.826769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.827557 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.827558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.827872 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.827873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.827983 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.828516 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:09.828530 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:09.830380 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:09.831733 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:09.831740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:09.831743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:09.831977 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:09.832206 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:09.832206 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:09.832207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:09.832316 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:09.832593 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:09.832600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:09.832765 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 13:17:09.832884 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 13:17:09.833066 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260926 13:17:09.833767 14636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260926 13:17:09.833864 14636 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:09.834025 14636 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260926 13:17:09.834420 14177 ts_manager.cc:194] Registered new tserver with Master: 433883ff49c34258b42293dcc698fde3 (127.0.98.3:32983)
I20260926 13:17:09.834755 14177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50211
I20260926 13:17:09.841240 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:17:09.848126 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.842532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.0.98.62:43329)
0926 13:17:09.842592 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.842594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.842715 (+ 121us) server_negotiation.cc:407] Connection header received
0926 13:17:09.842751 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.842752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.842760 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.842807 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:09.843304 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.843775 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.844468 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.844640 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.844691 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.845748 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:09.845749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:09.845749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:09.845755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:09.846999 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.847298 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.847299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.847299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.847348 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:09.847507 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:09.847508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:09.847509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:09.847667 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:09.847772 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.847908 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.848003 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:17:09.849579 14177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:17:09.851253 14177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:17:09.857611 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.853443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32983 (local address 127.0.0.1:53652)
0926 13:17:09.853699 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:09.853712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:09.853763 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:09.854084 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:09.854087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:09.854089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:09.854176 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.854179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.855181 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.855182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.856475 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.856477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.857395 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.857396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.857466 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.857470 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:09.857517 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:17:09.857548 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":57,"threads_started":1}
I20260926 13:17:09.857761 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.853620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38121 (local address 127.0.0.1:57608)
0926 13:17:09.853754 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:09.853763 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:09.853807 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:09.854315 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:09.854315 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:09.854317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:09.854373 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.854374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.855657 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.855658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.856631 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.856634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.857514 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.857515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.857589 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.857594 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:17:09.857652 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:17:09.857694 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":54,"threads_started":1}
I20260926 13:17:09.857889 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.853581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.0.98.3:32983)
0926 13:17:09.853924 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.853929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.853939 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:09.853978 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.853980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.853995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.854066 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:09.854254 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.855077 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.856561 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.857284 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.857393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.857468 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.857652 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.857812 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":85,"threads_started":1}
I20260926 13:17:09.857971 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.853527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40586 (local address 127.0.98.1:42297)
0926 13:17:09.853849 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.853853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.853863 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:09.853909 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.853910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.853923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.854023 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:09.854294 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.855432 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.856700 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.857201 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.857288 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.857349 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.857822 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.857868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":85,"threads_started":1}
I20260926 13:17:09.857602 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.853417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42297 (local address 127.0.0.1:40586)
0926 13:17:09.853548 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:09.853591 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:09.853659 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:09.854116 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:09.854117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:09.854119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:09.854203 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.854206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.855547 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.855548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.856614 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:09.856615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.857287 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.857288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:09.857371 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.857386 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:17:09.857453 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 13:17:09.857514 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260926 13:17:09.858357 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:09.853830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57608 (local address 127.0.98.2:38121)
0926 13:17:09.854031 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:09.854035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:09.854046 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:09.854152 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:09.854155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:09.854216 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:09.854453 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:09.854524 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.855543 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.856722 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:09.857416 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:09.857506 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:09.857558 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:09.857672 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 13:17:09.857714 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":106,"threads_started":1}
I20260926 13:17:09.858611 14570 tablet_service.cc:1467] Processing CreateTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 (DEFAULT_TABLE table=test-table [id=79aa7a0018a44cdaa0cb380ed8ca954f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:09.858757 14302 tablet_service.cc:1467] Processing CreateTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 (DEFAULT_TABLE table=test-table [id=79aa7a0018a44cdaa0cb380ed8ca954f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:09.858982 14570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a721f382ad4b4fd0af44a997a6a46d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:09.859046 14302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a721f382ad4b4fd0af44a997a6a46d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:09.859226 14436 tablet_service.cc:1467] Processing CreateTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 (DEFAULT_TABLE table=test-table [id=79aa7a0018a44cdaa0cb380ed8ca954f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:09.859467 14436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a721f382ad4b4fd0af44a997a6a46d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:09.862035 14650 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Bootstrap starting.
I20260926 13:17:09.862531 14652 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Bootstrap starting.
I20260926 13:17:09.862926 14650 tablet_bootstrap.cc:654] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:09.863199 14652 tablet_bootstrap.cc:654] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:09.863533 14650 log.cc:824] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:09.863798 14652 log.cc:824] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:09.864149 14651 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Bootstrap starting.
I20260926 13:17:09.864362 14650 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: No bootstrap required, opened a new log
I20260926 13:17:09.864457 14650 ts_tablet_manager.cc:1397] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:09.864795 14651 tablet_bootstrap.cc:654] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:09.865391 14651 log.cc:824] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:09.865928 14652 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: No bootstrap required, opened a new log
I20260926 13:17:09.866055 14652 ts_tablet_manager.cc:1397] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 13:17:09.866101 14651 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: No bootstrap required, opened a new log
I20260926 13:17:09.866207 14651 ts_tablet_manager.cc:1397] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:09.866724 14650 raft_consensus.cc:348] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:09.866919 14650 raft_consensus.cc:374] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:09.866959 14650 raft_consensus.cc:729] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485f154f73dc4a12bdf2032e94cee6c9, State: Initialized, Role: FOLLOWER
I20260926 13:17:09.867146 14650 consensus_queue.cc:260] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:09.867451 14650 ts_tablet_manager.cc:1428] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:17:09.867468 14502 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
I20260926 13:17:09.867925 14651 raft_consensus.cc:348] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:09.868104 14651 raft_consensus.cc:374] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:09.868139 14651 raft_consensus.cc:729] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433883ff49c34258b42293dcc698fde3, State: Initialized, Role: FOLLOWER
I20260926 13:17:09.868135 14652 raft_consensus.cc:348] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:09.868273 14651 consensus_queue.cc:260] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:09.868321 14652 raft_consensus.cc:374] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:09.868362 14652 raft_consensus.cc:729] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69313ddfd226493da7f9ee1628837030, State: Initialized, Role: FOLLOWER
I20260926 13:17:09.868508 14652 consensus_queue.cc:260] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:09.868592 14651 ts_tablet_manager.cc:1428] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:09.868640 14636 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
I20260926 13:17:09.868830 14652 ts_tablet_manager.cc:1428] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:17:09.868963 14368 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
W20260926 13:17:09.922431 14503 tablet.cc:2369] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:17:10.016923 14369 tablet.cc:2369] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:17:10.074385 14637 tablet.cc:2369] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:17:10.234153 14656 raft_consensus.cc:482] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:17:10.234339 14656 raft_consensus.cc:504] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.234671 14656 leader_election.cc:288] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297), 433883ff49c34258b42293dcc698fde3 (127.0.98.3:32983)
I20260926 13:17:10.238250 14647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.234974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59559 (local address 127.0.98.3:32983)
0926 13:17:10.235038 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.235040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.235126 (+ 86us) server_negotiation.cc:407] Connection header received
0926 13:17:10.235180 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.235181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.235193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.235266 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:10.235526 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.235975 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.237498 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.237855 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.237920 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.237928 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.238104 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.238148 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:17:10.238262 14659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.234874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32983 (local address 127.0.98.2:59559)
0926 13:17:10.235059 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.235070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.235119 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.235281 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.235282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.235284 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:10.235435 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.235437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.236081 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.236082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.237416 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.237420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.237949 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.237950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.238029 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.238035 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.238090 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.238131 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":133,"spinlock_wait_cycles":13184,"thread_start_us":73,"threads_started":1}
I20260926 13:17:10.238346 14646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.235029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35799 (local address 127.0.98.1:42297)
0926 13:17:10.235066 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.235068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.235117 (+ 49us) server_negotiation.cc:407] Connection header received
0926 13:17:10.235180 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.235181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.235193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.235266 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:10.235507 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.235976 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.237548 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.237945 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.238017 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.238028 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.238229 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.238275 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:17:10.238489 14660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.234874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42297 (local address 127.0.98.2:35799)
0926 13:17:10.235034 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.235046 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.235119 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.235281 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.235282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.235285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:10.235382 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.235384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.236061 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.236062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.237417 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.237420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.238037 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.238039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.238129 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.238133 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.238220 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.238263 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":55,"thread_start_us":81,"threads_started":1}
I20260926 13:17:10.238682 14590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a721f382ad4b4fd0af44a997a6a46d85" candidate_uuid: "485f154f73dc4a12bdf2032e94cee6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433883ff49c34258b42293dcc698fde3" is_pre_election: true
I20260926 13:17:10.238821 14590 raft_consensus.cc:2454] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 485f154f73dc4a12bdf2032e94cee6c9 in term 0.
I20260926 13:17:10.239034 14392 leader_election.cc:302] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433883ff49c34258b42293dcc698fde3, 485f154f73dc4a12bdf2032e94cee6c9; no voters:
I20260926 13:17:10.239008 14322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a721f382ad4b4fd0af44a997a6a46d85" candidate_uuid: "485f154f73dc4a12bdf2032e94cee6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69313ddfd226493da7f9ee1628837030" is_pre_election: true
I20260926 13:17:10.239146 14322 raft_consensus.cc:2454] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 485f154f73dc4a12bdf2032e94cee6c9 in term 0.
I20260926 13:17:10.239171 14656 raft_consensus.cc:2781] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:17:10.239274 14656 raft_consensus.cc:482] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:17:10.239310 14656 raft_consensus.cc:3037] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:10.240015 14656 raft_consensus.cc:504] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.240163 14656 leader_election.cc:288] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [CANDIDATE]: Term 1 election: Requested vote from peers 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297), 433883ff49c34258b42293dcc698fde3 (127.0.98.3:32983)
I20260926 13:17:10.240303 14590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a721f382ad4b4fd0af44a997a6a46d85" candidate_uuid: "485f154f73dc4a12bdf2032e94cee6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433883ff49c34258b42293dcc698fde3"
I20260926 13:17:10.240342 14322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a721f382ad4b4fd0af44a997a6a46d85" candidate_uuid: "485f154f73dc4a12bdf2032e94cee6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69313ddfd226493da7f9ee1628837030"
I20260926 13:17:10.240382 14590 raft_consensus.cc:3037] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:10.240432 14322 raft_consensus.cc:3037] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:10.241168 14590 raft_consensus.cc:2454] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485f154f73dc4a12bdf2032e94cee6c9 in term 1.
I20260926 13:17:10.241293 14322 raft_consensus.cc:2454] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 485f154f73dc4a12bdf2032e94cee6c9 in term 1.
I20260926 13:17:10.241320 14392 leader_election.cc:302] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433883ff49c34258b42293dcc698fde3, 485f154f73dc4a12bdf2032e94cee6c9; no voters:
I20260926 13:17:10.241400 14656 raft_consensus.cc:2781] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:10.241562 14656 raft_consensus.cc:686] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 485f154f73dc4a12bdf2032e94cee6c9, State: Running, Role: LEADER
I20260926 13:17:10.241677 14656 consensus_queue.cc:237] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.242558 14177 catalog_manager.cc:5495] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "485f154f73dc4a12bdf2032e94cee6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } health_report { overall_health: HEALTHY } } }
I20260926 13:17:10.297349 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.294922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.0.98.2:38121)
0926 13:17:10.294965 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.294967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.294977 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:10.295053 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.295054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.295074 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.295154 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:10.295302 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.295770 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.296524 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.296737 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.296814 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.297013 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:10.297117 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:17:10.297215 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.297235 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.297256 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:17:10.299261 14322 raft_consensus.cc:1261] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Refusing update from remote peer 485f154f73dc4a12bdf2032e94cee6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:10.299261 14590 raft_consensus.cc:1261] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Refusing update from remote peer 485f154f73dc4a12bdf2032e94cee6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:10.299623 14661 consensus_queue.cc:1035] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:10.299782 14656 consensus_queue.cc:1035] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:10.302546 14666 mvcc.cc:205] Tried to move back new op lower bound from 7333595669702754304 to 7333595669470560256. Current Snapshot: MvccSnapshot[applied={T|T < 7333595669702754304}]
I20260926 13:17:10.302867 14665 mvcc.cc:205] Tried to move back new op lower bound from 7333595669702754304 to 7333595669470560256. Current Snapshot: MvccSnapshot[applied={T|T < 7333595669702754304 or (T in {7333595669702754304})}]
I20260926 13:17:10.304493 14177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:10.304589 14177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:10.305332 14177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Sending DeleteTablet for 3 replicas of tablet a721f382ad4b4fd0af44a997a6a46d85
I20260926 13:17:10.305675 14570 tablet_service.cc:1514] Processing DeleteTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53652
I20260926 13:17:10.305708 14302 tablet_service.cc:1514] Processing DeleteTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40586
I20260926 13:17:10.305899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14239
I20260926 13:17:10.305950 14436 tablet_service.cc:1514] Processing DeleteTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57608
I20260926 13:17:10.307184 14674 tablet_replica.cc:330] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: stopping tablet replica
I20260926 13:17:10.307291 14674 raft_consensus.cc:2229] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:10.307397 14674 raft_consensus.cc:2258] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:10.307845 14674 ts_tablet_manager.cc:1905] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:10.309831 14674 ts_tablet_manager.cc:1918] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:10.309916 14674 log.cc:1196] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/a721f382ad4b4fd0af44a997a6a46d85
I20260926 13:17:10.310184 14674 ts_tablet_manager.cc:1939] T a721f382ad4b4fd0af44a997a6a46d85 P 433883ff49c34258b42293dcc698fde3: Deleting consensus metadata
I20260926 13:17:10.311371 14164 catalog_manager.cc:4841] TS 433883ff49c34258b42293dcc698fde3 (127.0.98.3:32983): tablet a721f382ad4b4fd0af44a997a6a46d85 (table test-table [id=79aa7a0018a44cdaa0cb380ed8ca954f]) successfully deleted
W20260926 13:17:10.312508 14163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42297 (error 108)
I20260926 13:17:10.313041 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.312893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42297 (local address 127.0.0.1:40598)
0926 13:17:10.312960 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.312999 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:17:10.313081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14373
W20260926 13:17:10.313215 14163 catalog_manager.cc:4584] TS 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a721f382ad4b4fd0af44a997a6a46d85: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42297: connect: Connection refused (error 111)
I20260926 13:17:10.318279 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.318136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38121 (local address 127.0.0.1:57626)
0926 13:17:10.318208 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.318239 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 13:17:10.318302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14507
I20260926 13:17:10.323215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14145
I20260926 13:17:10.328578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--webserver_interface=127.0.98.62
--webserver_port=33691
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:10.417837 14678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:10.418042 14678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:10.418063 14678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:10.419220 14678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:10.419265 14678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:10.419279 14678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:10.419327 14678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:10.419353 14678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:10.420800 14678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:10.421056 14678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:10.421283 14678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:10.429754 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.429761 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:10.429924 14678 server_base.cc:1034] running on GCE node
W20260926 13:17:10.429800 14684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:10.430377 14678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:10.430766 14678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:10.431906 14678 hybrid_clock.cc:648] HybridClock initialized: now 1790428630431887 us; error 34 us; skew 500 ppm
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:10.441089 14678 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:10.441664 14678 webserver.cc:460] Webserver started at http://127.0.98.62:33691/ using document root <none> and password file <none>
I20260926 13:17:10.441944 14678 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:10.442027 14678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:10.443573 14678 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:10.444234 14694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.444410 14678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:17:10.444514 14678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "e686222b064244fca0b4fa1cc7b253c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "fdc0adb125f802f04dcd201de58be578"
server_key_iv: "bb50ffefc37a04e9ce17beb0f79caea3"
server_key_version: "encryptionkey@0"
I20260926 13:17:10.444923 14678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:10.459600 14678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:10.461365 14678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:10.461522 14678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:10.465678 14678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43329
I20260926 13:17:10.465711 14746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43329 every 8 connection(s)
I20260926 13:17:10.466096 14678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:10.467772 14747 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:17:10.468485 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap starting.
I20260926 13:17:10.473965 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14678
I20260926 13:17:10.474339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42297
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=41187
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:10.477285 14747 log.cc:824] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:10.479354 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:10.479655 14747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap complete.
I20260926 13:17:10.481980 14747 raft_consensus.cc:348] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:10.482250 14747 raft_consensus.cc:729] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e686222b064244fca0b4fa1cc7b253c9, State: Initialized, Role: FOLLOWER
I20260926 13:17:10.482424 14747 consensus_queue.cc:260] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:10.482507 14747 raft_consensus.cc:388] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:10.482559 14747 raft_consensus.cc:482] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:10.482605 14747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:17:10.483671 14747 raft_consensus.cc:504] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:10.483755 14747 leader_election.cc:302] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e686222b064244fca0b4fa1cc7b253c9; no voters:
I20260926 13:17:10.483952 14747 leader_election.cc:288] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:17:10.483986 14751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:17:10.484100 14751 raft_consensus.cc:686] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 LEADER]: Becoming Leader. State: Replica: e686222b064244fca0b4fa1cc7b253c9, State: Running, Role: LEADER
I20260926 13:17:10.484184 14751 consensus_queue.cc:237] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:10.484221 14747 sys_catalog.cc:564] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:10.484433 14751 sys_catalog.cc:455] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e686222b064244fca0b4fa1cc7b253c9. Latest consensus state: current_term: 2 leader_uuid: "e686222b064244fca0b4fa1cc7b253c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260926 13:17:10.484510 14751 sys_catalog.cc:458] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:10.484437 14752 sys_catalog.cc:455] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e686222b064244fca0b4fa1cc7b253c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260926 13:17:10.484684 14752 sys_catalog.cc:458] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:10.485186 14755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:10.485935 14755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:10.486030 14755 catalog_manager.cc:1259] Loaded cluster ID: 2521171d6a304750b7bc1ef7d2559840
I20260926 13:17:10.486059 14755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:17:10.486316 14755 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 13:17:10.491365 14755 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 13:17:10.494351 14755 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:10.494529 14755 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Loaded TSK: 0
I20260926 13:17:10.494731 14755 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:17:10.565500 14749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:10.565804 14749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:10.565840 14749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:10.567512 14749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:10.567591 14749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:10.567637 14749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:17:10.569121 14749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41187
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:10.569411 14749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:10.569643 14749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:10.575587 14775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.575708 14774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.575590 14777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:10.575909 14749 server_base.cc:1034] running on GCE node
I20260926 13:17:10.576125 14749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:10.576395 14749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:10.577580 14749 hybrid_clock.cc:648] HybridClock initialized: now 1790428630577561 us; error 31 us; skew 500 ppm
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:10.580650 14749 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:17:10.581218 14749 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260926 13:17:10.581501 14749 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:10.581581 14749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:10.583374 14749 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:10.584151 14784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.584337 14749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:10.584424 14749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "69313ddfd226493da7f9ee1628837030"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "4f8cc71cd797c48d569a3ff56a9e4071"
server_key_iv: "ccdd52beab58f6900f983c6fb04ee6f2"
server_key_version: "encryptionkey@0"
I20260926 13:17:10.584787 14749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:10.597903 14749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:10.600307 14749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:10.600449 14749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:10.600696 14749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:10.601146 14791 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:17:10.603046 14749 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:17:10.603120 14749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:17:10.603170 14749 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:17:10.603758 14749 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:17:10.603811 14749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:17:10.603890 14791 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Bootstrap starting.
I20260926 13:17:10.610919 14749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42297
I20260926 13:17:10.610954 14898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42297 every 8 connection(s)
I20260926 13:17:10.611320 14749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:17:10.614423 14791 log.cc:824] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:10.616498 14791 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:10.616780 14791 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Bootstrap complete.
I20260926 13:17:10.618008 14791 ts_tablet_manager.cc:1397] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260926 13:17:10.619146 14791 raft_consensus.cc:348] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.619477 14791 raft_consensus.cc:729] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69313ddfd226493da7f9ee1628837030, State: Initialized, Role: FOLLOWER
I20260926 13:17:10.619619 14791 consensus_queue.cc:260] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.619869 14791 ts_tablet_manager.cc:1428] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260926 13:17:10.620124 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14749
I20260926 13:17:10.620404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38121
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=39409
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:10.625146 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.612070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33431 (local address 127.0.98.62:43329)
0926 13:17:10.614316 (+ 2246us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.614325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.614363 (+ 38us) server_negotiation.cc:407] Connection header received
0926 13:17:10.614493 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.614499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.614554 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.614716 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:10.615475 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.616797 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.618428 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.619027 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.619159 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.621366 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:10.621370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:10.621370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:10.621381 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:10.623425 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.623853 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.623856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.623856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.623953 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.624213 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.624214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.624214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.624431 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:10.624518 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.624848 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.624977 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":122,"threads_started":1}
I20260926 13:17:10.625156 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.611985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:33431)
0926 13:17:10.612160 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.612200 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.612621 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.614748 (+ 2127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.614751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.615100 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:10.615338 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.615344 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.616963 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.616964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.618335 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.618337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.619128 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.619129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.619254 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.619841 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:10.619850 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:10.621256 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:10.623522 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.623528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.623530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.623739 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.624026 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.624027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.624027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.624138 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.624511 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:10.624516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:10.624674 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.624797 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.624960 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260926 13:17:10.625824 14899 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260926 13:17:10.625934 14899 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:10.626137 14899 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260926 13:17:10.626827 14711 ts_manager.cc:194] Registered new tserver with Master: 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297)
I20260926 13:17:10.627843 14711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33431
I20260926 13:17:10.628693 14899 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
I20260926 13:17:10.633375 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.628162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.0.98.1:42297)
0926 13:17:10.630536 (+ 2374us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.630545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.630553 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:17:10.630577 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.630578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.630589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.630672 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:10.630816 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.631507 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.632469 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.632959 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.633056 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.633109 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.633262 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.633299 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2344,"thread_start_us":72,"threads_started":1}
I20260926 13:17:10.633373 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.627389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42297 (local address 127.0.0.1:40600)
0926 13:17:10.630289 (+ 2900us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.630347 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.630412 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.630678 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.630680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.630682 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:10.630743 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.630744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.631599 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.631600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.632359 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.632361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.633059 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.633060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.633141 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.633160 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.633247 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.633299 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2875,"thread_start_us":80,"threads_started":1}
I20260926 13:17:10.633929 14833 tablet_service.cc:1514] Processing DeleteTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40600
I20260926 13:17:10.635212 14908 tablet_replica.cc:330] stopping tablet replica
I20260926 13:17:10.635438 14908 raft_consensus.cc:2229] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:10.635515 14908 raft_consensus.cc:2258] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:10.635892 14908 ts_tablet_manager.cc:1905] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:10.638672 14908 ts_tablet_manager.cc:1918] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:10.638756 14908 log.cc:1196] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/a721f382ad4b4fd0af44a997a6a46d85
I20260926 13:17:10.639020 14908 ts_tablet_manager.cc:1939] T a721f382ad4b4fd0af44a997a6a46d85 P 69313ddfd226493da7f9ee1628837030: Deleting consensus metadata
I20260926 13:17:10.639523 14697 catalog_manager.cc:4841] TS 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297): tablet a721f382ad4b4fd0af44a997a6a46d85 (table test-table [id=79aa7a0018a44cdaa0cb380ed8ca954f]) successfully deleted
W20260926 13:17:10.711941 14905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:10.712182 14905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:10.712231 14905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:10.713433 14905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:10.713506 14905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:10.713551 14905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:10.715536 14905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39409
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:10.715802 14905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:10.716080 14905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:10.721421 14914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.721457 14913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.721424 14916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:10.721607 14905 server_base.cc:1034] running on GCE node
I20260926 13:17:10.721848 14905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:10.722093 14905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:10.723294 14905 hybrid_clock.cc:648] HybridClock initialized: now 1790428630723292 us; error 54 us; skew 500 ppm
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:10.725765 14905 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:10.726315 14905 webserver.cc:460] Webserver started at http://127.0.98.2:39409/ using document root <none> and password file <none>
I20260926 13:17:10.726564 14905 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:10.726653 14905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:10.727994 14905 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:10.728668 14923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.728849 14905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:17:10.728947 14905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "485f154f73dc4a12bdf2032e94cee6c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "3d3eed8a7586df6aa0bcf7d9c12e5c5a"
server_key_iv: "6fcab3809b89975577f6dc4308f13c18"
server_key_version: "encryptionkey@0"
I20260926 13:17:10.729303 14905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:10.744949 14905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:10.746971 14905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:10.747145 14905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:10.747378 14905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:10.747817 14930 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:17:10.749330 14905 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:17:10.749392 14905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:17:10.749445 14905 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:17:10.750063 14905 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:17:10.750118 14905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:17:10.750216 14930 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Bootstrap starting.
I20260926 13:17:10.757248 14905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38121
I20260926 13:17:10.757335 15037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38121 every 8 connection(s)
I20260926 13:17:10.757688 14905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:17:10.759006 14930 log.cc:824] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:10.761409 14930 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:10.761703 14930 tablet_bootstrap.cc:492] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Bootstrap complete.
I20260926 13:17:10.762830 14930 ts_tablet_manager.cc:1397] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260926 13:17:10.764256 14930 raft_consensus.cc:348] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.764487 14930 raft_consensus.cc:729] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485f154f73dc4a12bdf2032e94cee6c9, State: Initialized, Role: FOLLOWER
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:10.764622 14930 consensus_queue.cc:260] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.764955 14930 ts_tablet_manager.cc:1428] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:17:10.766637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 14905
I20260926 13:17:10.766940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32983
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=36401
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:10.768415 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.758492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59209 (local address 127.0.98.62:43329)
0926 13:17:10.758567 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.758569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.758812 (+ 243us) server_negotiation.cc:407] Connection header received
0926 13:17:10.759475 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.759476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.759491 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.759557 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:10.760323 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.760817 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.762376 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.762617 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.762705 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.765249 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:10.765250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:10.765250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:10.765259 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:10.766944 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.767403 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.767404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.767405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.767476 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.767709 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.767709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.767710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.767862 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:10.767942 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.768222 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.768331 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:17:10.768651 15040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.758390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:59209)
0926 13:17:10.758661 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.758708 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.759347 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.759621 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.759626 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.759971 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:10.760229 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.760236 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.760920 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.760922 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.762281 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.762284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.762714 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.762715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.762869 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.763415 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:10.763425 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:10.765152 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:10.767035 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.767043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.767046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.767323 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.767547 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.767548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.767548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.767645 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.767943 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:10.767948 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:10.768061 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.768181 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.768376 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
I20260926 13:17:10.769255 15038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260926 13:17:10.769361 15038 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:10.769544 15038 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260926 13:17:10.770108 14711 ts_manager.cc:194] Registered new tserver with Master: 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2:38121)
I20260926 13:17:10.770784 14711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59209
I20260926 13:17:10.771528 15038 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 13:17:10.779112 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.770612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.98.2:38121)
0926 13:17:10.770754 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.770758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.770768 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:10.770807 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.770809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.770822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.770945 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:10.771536 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.772392 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.773459 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.773736 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.773788 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.776298 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:10.776305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:10.776306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:10.776313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:10.777884 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.778354 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.778357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.778357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.778412 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.778626 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.778627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.778627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.778757 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:10.778815 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.778979 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.779049 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260926 13:17:10.779145 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.770445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38121 (local address 127.0.0.1:57632)
0926 13:17:10.770498 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.770508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.770547 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.770978 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.770979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.771387 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:10.771439 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.771442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.772473 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.772474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.773368 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.773372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.773812 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.773816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.773886 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.774488 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:10.774494 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:10.776193 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:10.777967 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.777975 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.777977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.778280 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.778490 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.778491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.778491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.778557 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.778816 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:10.778822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:10.778873 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.778956 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.779052 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:17:10.779721 14972 tablet_service.cc:1514] Processing DeleteTablet for tablet a721f382ad4b4fd0af44a997a6a46d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57632
I20260926 13:17:10.780328 15045 tablet_replica.cc:330] stopping tablet replica
I20260926 13:17:10.780560 15045 raft_consensus.cc:2229] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:10.780709 15045 raft_consensus.cc:2258] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:10.781013 15045 ts_tablet_manager.cc:1905] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:10.783324 15045 ts_tablet_manager.cc:1918] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:10.783401 15045 log.cc:1196] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/a721f382ad4b4fd0af44a997a6a46d85
I20260926 13:17:10.783649 15045 ts_tablet_manager.cc:1939] T a721f382ad4b4fd0af44a997a6a46d85 P 485f154f73dc4a12bdf2032e94cee6c9: Deleting consensus metadata
I20260926 13:17:10.784170 14697 catalog_manager.cc:4841] TS 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2:38121): tablet a721f382ad4b4fd0af44a997a6a46d85 (table test-table [id=79aa7a0018a44cdaa0cb380ed8ca954f]) successfully deleted
W20260926 13:17:10.850111 15043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:10.850364 15043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:10.850414 15043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:10.851549 15043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:10.851616 15043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:10.851661 15043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:10.853083 15043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36401
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:10.853348 15043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:10.853577 15043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:10.858407 15050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.858383 15053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:10.858429 15051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:10.858551 15043 server_base.cc:1034] running on GCE node
I20260926 13:17:10.858747 15043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:10.858946 15043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:10.860083 15043 hybrid_clock.cc:648] HybridClock initialized: now 1790428630860066 us; error 31 us; skew 500 ppm
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:10.862543 15043 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:10.862990 15043 webserver.cc:460] Webserver started at http://127.0.98.3:36401/ using document root <none> and password file <none>
I20260926 13:17:10.863240 15043 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:10.863323 15043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:10.864621 15043 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:10.865202 15060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.865363 15043 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.865465 15043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "433883ff49c34258b42293dcc698fde3"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "84b7f60923555d0f3c684d2ff910c462"
server_key_iv: "a847124a38b791e017b0265d1f98d175"
server_key_version: "encryptionkey@0"
I20260926 13:17:10.865811 15043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:10.878159 15043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:10.880584 15043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:10.880748 15043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:10.880987 15043 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:10.881299 15043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:10.881352 15043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.881392 15043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:10.881420 15043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:10.887954 15043 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32983
I20260926 13:17:10.888010 15173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32983 every 8 connection(s)
I20260926 13:17:10.888409 15043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:17:10.892180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15043
Sep 26 13:17:10 dist-test-slave-f864 krb5kdc[14132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:10.898290 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.889139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38855 (local address 127.0.98.62:43329)
0926 13:17:10.889202 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.889204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.889387 (+ 183us) server_negotiation.cc:407] Connection header received
0926 13:17:10.890371 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.890372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.890382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.890432 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:10.890994 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.891480 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.892537 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.892833 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.892911 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.895205 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:10.895206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:10.895206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:10.895214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:10.896899 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.897342 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.897343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.897344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.897398 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.897653 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.897654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.897654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.897761 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:10.897832 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.898096 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.898202 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:17:10.898497 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.889140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:38855)
0926 13:17:10.889286 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.889321 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.890282 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.890496 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.890499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.890727 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:10.890912 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.890915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.891589 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.891589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.892444 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.892446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.892917 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.892918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.893025 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.893587 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:10.893597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:10.895094 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:10.896992 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.896997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.896999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.897203 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.897476 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:10.897477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:10.897477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:10.897577 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:10.897835 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:10.897841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:10.897967 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.898074 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.898253 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260926 13:17:10.898962 15174 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43329
I20260926 13:17:10.899075 15174 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:10.899242 15174 heartbeater.cc:511] Master 127.0.98.62:43329 requested a full tablet report, sending...
I20260926 13:17:10.899621 14711 ts_manager.cc:194] Registered new tserver with Master: 433883ff49c34258b42293dcc698fde3 (127.0.98.3:32983)
I20260926 13:17:10.900017 14711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38855
I20260926 13:17:10.900503 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.892534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.0.98.62:43329)
0926 13:17:10.892634 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.892636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.892669 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:17:10.892742 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.892743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.892752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.892801 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:10.893486 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.894231 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.895695 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.895880 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.895961 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.897232 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:10.897233 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:10.897233 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:10.897240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:10.898845 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.899282 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.899284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.899284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.899341 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:10.899534 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:10.899535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:10.899536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:10.899654 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:10.899805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.900323 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.900402 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
I20260926 13:17:10.901268 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:17:10.903956 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.901648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.0.98.62:43329)
0926 13:17:10.901697 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.901699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.901810 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:17:10.901878 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.901879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.901905 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.901970 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:10.902061 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.902558 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.903270 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.903449 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.903499 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.903678 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:10.903761 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:17:10.903820 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.903891 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.903907 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:17:10.904193 14711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:17:10.905797 14711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:17:10.907869 14833 tablet_service.cc:1467] Processing CreateTablet for tablet 25587f9be217406386b1fb140bed3ad6 (DEFAULT_TABLE table=test-table [id=147cea4238e44a3c84a4c5c7cc0cd88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:10.907893 14972 tablet_service.cc:1467] Processing CreateTablet for tablet 25587f9be217406386b1fb140bed3ad6 (DEFAULT_TABLE table=test-table [id=147cea4238e44a3c84a4c5c7cc0cd88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:10.908103 14972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25587f9be217406386b1fb140bed3ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:10.908097 14833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25587f9be217406386b1fb140bed3ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:10.909572 14791 tablet_bootstrap.cc:492] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Bootstrap starting.
I20260926 13:17:10.909670 14930 tablet_bootstrap.cc:492] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Bootstrap starting.
I20260926 13:17:10.909968 14791 tablet_bootstrap.cc:654] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:10.910002 14930 tablet_bootstrap.cc:654] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:10.910537 14791 tablet_bootstrap.cc:492] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: No bootstrap required, opened a new log
I20260926 13:17:10.910576 14791 ts_tablet_manager.cc:1397] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:10.910738 14791 raft_consensus.cc:348] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.910830 14791 raft_consensus.cc:374] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:10.910885 14791 raft_consensus.cc:729] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69313ddfd226493da7f9ee1628837030, State: Initialized, Role: FOLLOWER
I20260926 13:17:10.910939 14791 consensus_queue.cc:260] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.911031 14791 ts_tablet_manager.cc:1428] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 13:17:10.911401 14930 tablet_bootstrap.cc:492] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: No bootstrap required, opened a new log
I20260926 13:17:10.911453 14930 ts_tablet_manager.cc:1397] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:17:10.911856 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.907621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32983 (local address 127.0.0.1:53654)
0926 13:17:10.907662 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.907673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.907708 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.908198 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.908199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.908201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:10.908258 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.908259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.909509 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.909510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.910625 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.910629 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.911648 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.911650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.911715 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.911719 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.911776 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.911809 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:17:10.911825 14930 raft_consensus.cc:348] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.911893 14930 raft_consensus.cc:374] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:10.911935 14930 raft_consensus.cc:729] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 485f154f73dc4a12bdf2032e94cee6c9, State: Initialized, Role: FOLLOWER
I20260926 13:17:10.911939 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.907712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53654 (local address 127.0.98.3:32983)
0926 13:17:10.907919 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.907924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.907946 (+ 22us) server_negotiation.cc:407] Connection header received
0926 13:17:10.907984 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.907986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.908003 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.908121 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:10.908354 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.909369 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.910717 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.911483 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.911555 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.911618 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.911806 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.911855 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":114,"threads_started":1}
I20260926 13:17:10.911988 14930 consensus_queue.cc:260] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.912087 14930 ts_tablet_manager.cc:1428] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:10.912580 15108 tablet_service.cc:1467] Processing CreateTablet for tablet 25587f9be217406386b1fb140bed3ad6 (DEFAULT_TABLE table=test-table [id=147cea4238e44a3c84a4c5c7cc0cd88c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:10.912899 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25587f9be217406386b1fb140bed3ad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:10.915575 15183 tablet_bootstrap.cc:492] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Bootstrap starting.
I20260926 13:17:10.916142 15183 tablet_bootstrap.cc:654] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:10.916733 15183 log.cc:824] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:10.917439 15183 tablet_bootstrap.cc:492] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: No bootstrap required, opened a new log
I20260926 13:17:10.917524 15183 ts_tablet_manager.cc:1397] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:17:10.918957 15183 raft_consensus.cc:348] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.919135 15183 raft_consensus.cc:374] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:10.919180 15183 raft_consensus.cc:729] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433883ff49c34258b42293dcc698fde3, State: Initialized, Role: FOLLOWER
I20260926 13:17:10.919319 15183 consensus_queue.cc:260] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.919613 15183 ts_tablet_manager.cc:1428] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:17:10.919647 15174 heartbeater.cc:503] Master 127.0.98.62:43329 was elected leader, sending a full tablet report...
I20260926 13:17:10.970741 15185 raft_consensus.cc:482] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:17:10.970863 15185 raft_consensus.cc:504] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.971179 15185 leader_election.cc:288] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297), 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2:38121)
I20260926 13:17:10.974308 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.971458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51115 (local address 127.0.98.1:42297)
0926 13:17:10.971501 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.971503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.971516 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:17:10.971595 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.971596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.971609 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.971705 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:10.971895 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.972471 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.973461 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.973887 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.973943 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.973951 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.974134 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.974212 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:17:10.974313 15176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.971384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42297 (local address 127.0.98.3:51115)
0926 13:17:10.971460 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.971472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.971517 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.971716 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.971717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.971719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:10.971807 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.971809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.972561 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.972562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.973359 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.973361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.973975 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.973976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.974061 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.974066 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.974118 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.974200 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 13:17:10.974444 15186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.971502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38121 (local address 127.0.98.3:33765)
0926 13:17:10.971728 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:10.971740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:10.971801 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:10.971958 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:10.971959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:10.971961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:10.972026 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.972028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.972708 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.972709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.973465 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:10.973467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.974119 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.974120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:10.974232 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.974236 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:10.974293 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:17:10.974333 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":37,"thread_start_us":150,"threads_started":1}
I20260926 13:17:10.974476 15044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.971573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33765 (local address 127.0.98.2:38121)
0926 13:17:10.971631 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.971633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.971804 (+ 171us) server_negotiation.cc:407] Connection header received
0926 13:17:10.971840 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.971841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.971872 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.971951 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:10.972098 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.972619 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.973555 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.974011 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.974085 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.974197 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.974342 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.974398 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:17:10.974699 14992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25587f9be217406386b1fb140bed3ad6" candidate_uuid: "433883ff49c34258b42293dcc698fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485f154f73dc4a12bdf2032e94cee6c9" is_pre_election: true
I20260926 13:17:10.974712 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25587f9be217406386b1fb140bed3ad6" candidate_uuid: "433883ff49c34258b42293dcc698fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69313ddfd226493da7f9ee1628837030" is_pre_election: true
I20260926 13:17:10.974864 14992 raft_consensus.cc:2454] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433883ff49c34258b42293dcc698fde3 in term 0.
I20260926 13:17:10.974864 14853 raft_consensus.cc:2454] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 433883ff49c34258b42293dcc698fde3 in term 0.
I20260926 13:17:10.975095 15063 leader_election.cc:302] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433883ff49c34258b42293dcc698fde3, 485f154f73dc4a12bdf2032e94cee6c9; no voters:
I20260926 13:17:10.975243 15185 raft_consensus.cc:2781] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:17:10.975329 15185 raft_consensus.cc:482] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:17:10.975365 15185 raft_consensus.cc:3037] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:10.976070 15185 raft_consensus.cc:504] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.976212 15185 leader_election.cc:288] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [CANDIDATE]: Term 1 election: Requested vote from peers 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297), 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2:38121)
I20260926 13:17:10.976434 14992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25587f9be217406386b1fb140bed3ad6" candidate_uuid: "433883ff49c34258b42293dcc698fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "485f154f73dc4a12bdf2032e94cee6c9"
I20260926 13:17:10.976426 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25587f9be217406386b1fb140bed3ad6" candidate_uuid: "433883ff49c34258b42293dcc698fde3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69313ddfd226493da7f9ee1628837030"
I20260926 13:17:10.976509 14992 raft_consensus.cc:3037] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:10.976511 14853 raft_consensus.cc:3037] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:10.977471 14992 raft_consensus.cc:2454] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433883ff49c34258b42293dcc698fde3 in term 1.
I20260926 13:17:10.977638 15063 leader_election.cc:302] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433883ff49c34258b42293dcc698fde3, 485f154f73dc4a12bdf2032e94cee6c9; no voters:
I20260926 13:17:10.977758 15185 raft_consensus.cc:2781] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:10.977807 14853 raft_consensus.cc:2454] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 433883ff49c34258b42293dcc698fde3 in term 1.
I20260926 13:17:10.977970 15185 raft_consensus.cc:686] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 1 LEADER]: Becoming Leader. State: Replica: 433883ff49c34258b42293dcc698fde3, State: Running, Role: LEADER
I20260926 13:17:10.978072 15185 consensus_queue.cc:237] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } }
I20260926 13:17:10.979032 14711 catalog_manager.cc:5495] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 433883ff49c34258b42293dcc698fde3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "433883ff49c34258b42293dcc698fde3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433883ff49c34258b42293dcc698fde3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 } health_report { overall_health: UNKNOWN } } }
I20260926 13:17:10.988258 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:10.986058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.0.98.3:32983)
0926 13:17:10.986101 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:10.986103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:10.986110 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:17:10.986204 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:10.986205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:10.986235 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:10.986299 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:10.986440 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.986902 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.987610 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:10.987824 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:10.987873 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:10.988013 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:10.988120 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:17:10.988177 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:10.988190 (+ 13us) server_negotiation.cc:299] Negotiation successful
0926 13:17:10.988205 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:17:10.989948 14853 raft_consensus.cc:1261] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Refusing update from remote peer 433883ff49c34258b42293dcc698fde3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:10.990000 14992 raft_consensus.cc:1261] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Refusing update from remote peer 433883ff49c34258b42293dcc698fde3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:10.990383 15187 consensus_queue.cc:1035] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69313ddfd226493da7f9ee1628837030" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:10.990507 15185 consensus_queue.cc:1035] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "485f154f73dc4a12bdf2032e94cee6c9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:10.995318 14710 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:10.995399 14710 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:10.996075 14710 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Sending DeleteTablet for 3 replicas of tablet 25587f9be217406386b1fb140bed3ad6
I20260926 13:17:10.996356 14833 tablet_service.cc:1514] Processing DeleteTablet for tablet 25587f9be217406386b1fb140bed3ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40600
I20260926 13:17:10.996341 15108 tablet_service.cc:1514] Processing DeleteTablet for tablet 25587f9be217406386b1fb140bed3ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53654
I20260926 13:17:10.996399 14972 tablet_service.cc:1514] Processing DeleteTablet for tablet 25587f9be217406386b1fb140bed3ad6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57632
I20260926 13:17:10.996470 15045 tablet_replica.cc:330] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: stopping tablet replica
I20260926 13:17:10.996469 14908 tablet_replica.cc:330] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: stopping tablet replica
I20260926 13:17:10.996512 15045 raft_consensus.cc:2229] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:10.996512 14908 raft_consensus.cc:2229] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:10.996557 15045 raft_consensus.cc:2258] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:10.996568 14908 raft_consensus.cc:2258] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:10.996865 15045 ts_tablet_manager.cc:1905] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:10.996896 15200 tablet_replica.cc:330] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: stopping tablet replica
I20260926 13:17:10.997001 15200 raft_consensus.cc:2229] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:17:10.997148 15200 raft_consensus.cc:2258] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:10.997589 15200 ts_tablet_manager.cc:1905] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:10.998538 14908 ts_tablet_manager.cc:1905] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:10.998665 15045 ts_tablet_manager.cc:1918] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:10.998716 15045 log.cc:1196] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/25587f9be217406386b1fb140bed3ad6
I20260926 13:17:10.998942 15045 ts_tablet_manager.cc:1939] T 25587f9be217406386b1fb140bed3ad6 P 485f154f73dc4a12bdf2032e94cee6c9: Deleting consensus metadata
I20260926 13:17:10.999248 14697 catalog_manager.cc:4841] TS 485f154f73dc4a12bdf2032e94cee6c9 (127.0.98.2:38121): tablet 25587f9be217406386b1fb140bed3ad6 (table test-table [id=147cea4238e44a3c84a4c5c7cc0cd88c]) successfully deleted
I20260926 13:17:10.999410 15200 ts_tablet_manager.cc:1918] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:10.999483 15200 log.cc:1196] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/25587f9be217406386b1fb140bed3ad6
I20260926 13:17:10.999747 15200 ts_tablet_manager.cc:1939] T 25587f9be217406386b1fb140bed3ad6 P 433883ff49c34258b42293dcc698fde3: Deleting consensus metadata
I20260926 13:17:11.000057 14908 ts_tablet_manager.cc:1918] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:11.000119 14908 log.cc:1196] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/25587f9be217406386b1fb140bed3ad6
I20260926 13:17:11.000310 14908 ts_tablet_manager.cc:1939] T 25587f9be217406386b1fb140bed3ad6 P 69313ddfd226493da7f9ee1628837030: Deleting consensus metadata
I20260926 13:17:11.000644 14698 catalog_manager.cc:4841] TS 433883ff49c34258b42293dcc698fde3 (127.0.98.3:32983): tablet 25587f9be217406386b1fb140bed3ad6 (table test-table [id=147cea4238e44a3c84a4c5c7cc0cd88c]) successfully deleted
I20260926 13:17:11.000710 14697 catalog_manager.cc:4841] TS 69313ddfd226493da7f9ee1628837030 (127.0.98.1:42297): tablet 25587f9be217406386b1fb140bed3ad6 (table test-table [id=147cea4238e44a3c84a4c5c7cc0cd88c]) successfully deleted
I20260926 13:17:11.000802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14749
I20260926 13:17:11.007309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14905
I20260926 13:17:11.012590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15043
I20260926 13:17:11.017263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 14678
I20260926 13:17:11.022405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--webserver_interface=127.0.98.62
--webserver_port=33691
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:11.102696 15201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:11.102902 15201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:11.102932 15201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:11.103987 15201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:11.104018 15201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:11.104032 15201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:11.104045 15201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:11.104056 15201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:11.105429 15201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:11.105609 15201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:11.105770 15201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:11.110608 15206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.110633 15209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:11.110675 15201 server_base.cc:1034] running on GCE node
W20260926 13:17:11.110656 15207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:11.111101 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:11.111382 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:11.112546 15201 hybrid_clock.cc:648] HybridClock initialized: now 1790428631112550 us; error 46 us; skew 500 ppm
Sep 26 13:17:11 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:11.115113 15201 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:11.115597 15201 webserver.cc:460] Webserver started at http://127.0.98.62:33691/ using document root <none> and password file <none>
I20260926 13:17:11.115824 15201 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:11.115891 15201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:11.117226 15201 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:11.117906 15216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.118103 15201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:11.118237 15201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "e686222b064244fca0b4fa1cc7b253c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "fdc0adb125f802f04dcd201de58be578"
server_key_iv: "bb50ffefc37a04e9ce17beb0f79caea3"
server_key_version: "encryptionkey@0"
I20260926 13:17:11.118582 15201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:11.130251 15201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:11.132525 15201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:11.132680 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:11.136755 15201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43329
I20260926 13:17:11.136792 15268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43329 every 8 connection(s)
I20260926 13:17:11.137148 15201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:11.138868 15269 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:17:11.139556 15269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap starting.
I20260926 13:17:11.147058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15201
I20260926 13:17:11.147421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42297
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=41187
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:11.148948 15269 log.cc:824] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:11.151680 15269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:11.152002 15269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9: Bootstrap complete.
I20260926 13:17:11.154495 15269 raft_consensus.cc:348] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:11.154731 15269 raft_consensus.cc:729] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e686222b064244fca0b4fa1cc7b253c9, State: Initialized, Role: FOLLOWER
I20260926 13:17:11.154906 15269 consensus_queue.cc:260] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:11.154987 15269 raft_consensus.cc:388] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:11.155062 15269 raft_consensus.cc:482] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:11.155140 15269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:17:11.156220 15269 raft_consensus.cc:504] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:11.156311 15269 leader_election.cc:302] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e686222b064244fca0b4fa1cc7b253c9; no voters:
I20260926 13:17:11.156520 15269 leader_election.cc:288] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:17:11.156558 15273 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:17:11.156728 15273 raft_consensus.cc:686] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [term 3 LEADER]: Becoming Leader. State: Replica: e686222b064244fca0b4fa1cc7b253c9, State: Running, Role: LEADER
I20260926 13:17:11.156813 15269 sys_catalog.cc:564] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:11.156852 15273 consensus_queue.cc:237] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } }
I20260926 13:17:11.157191 15273 sys_catalog.cc:455] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e686222b064244fca0b4fa1cc7b253c9. Latest consensus state: current_term: 3 leader_uuid: "e686222b064244fca0b4fa1cc7b253c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260926 13:17:11.157196 15274 sys_catalog.cc:455] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e686222b064244fca0b4fa1cc7b253c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e686222b064244fca0b4fa1cc7b253c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43329 } } }
I20260926 13:17:11.157377 15274 sys_catalog.cc:458] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:11.157290 15273 sys_catalog.cc:458] T 00000000000000000000000000000000 P e686222b064244fca0b4fa1cc7b253c9 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:11.157647 15282 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:11.158468 15282 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:11.158592 15282 catalog_manager.cc:1259] Loaded cluster ID: 2521171d6a304750b7bc1ef7d2559840
I20260926 13:17:11.158874 15282 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 13:17:11.159114 15282 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ffb0335a9ad google::LogMessage::Fail() at ??:0
@ 0x7ffb0335d88a google::LogMessage::SendToLog() at ??:0
@ 0x7ffb0335a4de google::LogMessage::Flush() at ??:0
@ 0x7ffb0335ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffb0b27b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ffb0b2848b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ffb052280d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffb0521d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffb060816db start_thread at ??:0
@ 0x7ffb01e5861f clone at ??:0
W20260926 13:17:11.240649 15271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:11.240878 15271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:11.240924 15271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:11.242208 15271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:11.242277 15271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:11.242321 15271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:17:11.243875 15271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41187
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:11.244131 15271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:11.244393 15271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:11.249898 15306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.249969 15308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.249931 15305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:11.250223 15271 server_base.cc:1034] running on GCE node
I20260926 13:17:11.250483 15271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:11.250727 15271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:11.251885 15271 hybrid_clock.cc:648] HybridClock initialized: now 1790428631251871 us; error 36 us; skew 500 ppm
Sep 26 13:17:11 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:11.254412 15271 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:17:11.254848 15271 webserver.cc:460] Webserver started at http://127.0.98.1:41187/ using document root <none> and password file <none>
I20260926 13:17:11.255090 15271 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:11.255168 15271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:11.256522 15271 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:11.257167 15315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.257361 15271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:11.257473 15271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "69313ddfd226493da7f9ee1628837030"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "4f8cc71cd797c48d569a3ff56a9e4071"
server_key_iv: "ccdd52beab58f6900f983c6fb04ee6f2"
server_key_version: "encryptionkey@0"
I20260926 13:17:11.257822 15271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:11.265703 15271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:11.267937 15271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:11.268113 15271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:11.268328 15271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:11.268622 15271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:11.268676 15271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.268718 15271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:11.268745 15271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.275113 15271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42297
I20260926 13:17:11.275163 15428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42297 every 8 connection(s)
I20260926 13:17:11.275489 15271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:17:11.282974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15271
I20260926 13:17:11.283299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38121
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=39409
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:11.380193 15431 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:17:11.276200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:53171)
0926 13:17:11.276366 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.276400 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:11.276835 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:11.379679 (+102844us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
W20260926 13:17:11.380470 15429 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43329 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43329: Client connection negotiation failed: client connection to 127.0.98.62:43329: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:17:11.380836 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.380715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:35491)
0926 13:17:11.380771 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.380816 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:17:11.381175 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.381085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:57717)
0926 13:17:11.381129 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.381152 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:17:11.381275 15429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:17:11.384002 15432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:11.384217 15432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:11.384268 15432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:11.385453 15432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:11.385514 15432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:11.385573 15432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:11.387100 15432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39409
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:11.387337 15432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:11.387569 15432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:11.392585 15438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.392633 15437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.392633 15440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:11.392725 15432 server_base.cc:1034] running on GCE node
I20260926 13:17:11.393008 15432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:11.393251 15432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:11.394410 15432 hybrid_clock.cc:648] HybridClock initialized: now 1790428631394395 us; error 45 us; skew 500 ppm
Sep 26 13:17:11 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:11.397111 15432 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:11.397558 15432 webserver.cc:460] Webserver started at http://127.0.98.2:39409/ using document root <none> and password file <none>
I20260926 13:17:11.397748 15432 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:11.397789 15432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:11.399065 15432 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:11.399582 15447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.399756 15432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.399819 15432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "485f154f73dc4a12bdf2032e94cee6c9"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "3d3eed8a7586df6aa0bcf7d9c12e5c5a"
server_key_iv: "6fcab3809b89975577f6dc4308f13c18"
server_key_version: "encryptionkey@0"
I20260926 13:17:11.400134 15432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:11.411260 15432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:11.413010 15432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:11.413120 15432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:11.413303 15432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:11.413605 15432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:11.413638 15432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.413661 15432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:11.413708 15432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.419657 15432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38121
I20260926 13:17:11.419705 15560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38121 every 8 connection(s)
I20260926 13:17:11.420033 15432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:17:11.421005 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.420650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:42053)
0926 13:17:11.420796 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.420867 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
W20260926 13:17:11.421408 15561 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43329 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43329: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260926 13:17:11.421692 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.421606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:58673)
0926 13:17:11.421650 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.421676 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 13:17:11.421963 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.421889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:35287)
0926 13:17:11.421930 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.421951 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:17:11.422073 15561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:17:11.429184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15432
I20260926 13:17:11.429454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32983
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43329
--webserver_port=36401
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:11.510365 15564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:11.510571 15564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:11.510596 15564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:11.511754 15564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:11.511796 15564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:11.511822 15564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:11.513247 15564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32983
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36401
--tserver_master_addrs=127.0.98.62:43329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:11.513461 15564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:11.513679 15564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:11.518790 15570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.518914 15569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:11.518803 15572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:11.518836 15564 server_base.cc:1034] running on GCE node
I20260926 13:17:11.519140 15564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:11.519388 15564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:11.520562 15564 hybrid_clock.cc:648] HybridClock initialized: now 1790428631520546 us; error 44 us; skew 500 ppm
Sep 26 13:17:11 dist-test-slave-f864 krb5kdc[14132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:11.523075 15564 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:11.523449 15564 webserver.cc:460] Webserver started at http://127.0.98.3:36401/ using document root <none> and password file <none>
I20260926 13:17:11.523617 15564 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:11.523656 15564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:11.524874 15564 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:11.525403 15579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.525592 15564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:17:11.525660 15564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "433883ff49c34258b42293dcc698fde3"
format_stamp: "Formatted at 2026-09-26 13:17:09 on dist-test-slave-f864"
server_key: "84b7f60923555d0f3c684d2ff910c462"
server_key_iv: "a847124a38b791e017b0265d1f98d175"
server_key_version: "encryptionkey@0"
I20260926 13:17:11.525977 15564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:11.536412 15564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:11.538100 15564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:11.538270 15564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:11.538478 15564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:11.538779 15564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:11.538810 15564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.538877 15564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:11.538916 15564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:11.544683 15564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32983
I20260926 13:17:11.544757 15692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32983 every 8 connection(s)
I20260926 13:17:11.545096 15564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:17:11.546087 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.545732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:47983)
0926 13:17:11.545879 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.545954 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
W20260926 13:17:11.546475 15693 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43329 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43329: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
I20260926 13:17:11.546756 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.546660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:46613)
0926 13:17:11.546709 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.546740 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:17:11.547019 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:11.546936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:43579)
0926 13:17:11.546976 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:11.546997 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 13:17:11.547113 15693 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:17:11.554467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15564
I20260926 13:17:12.381879 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:12.381656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:50159)
0926 13:17:12.381781 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:12.381812 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260926 13:17:12.422577 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:12.422384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:41465)
0926 13:17:12.422491 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:12.422524 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260926 13:17:12.547629 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:12.547417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:58647)
0926 13:17:12.547552 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:12.547581 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260926 13:17:13.382598 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:13.382355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:42399)
0926 13:17:13.382495 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:13.382532 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260926 13:17:13.423305 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:13.423091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:44563)
0926 13:17:13.423229 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:13.423264 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260926 13:17:13.548309 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:13.548097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:35223)
0926 13:17:13.548236 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:13.548271 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260926 13:17:14.383282 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:14.383068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:48173)
0926 13:17:14.383195 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:14.383235 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260926 13:17:14.424002 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:14.423801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:50081)
0926 13:17:14.423925 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:14.423963 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260926 13:17:14.548925 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:14.548734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:44755)
0926 13:17:14.548849 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:14.548887 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260926 13:17:15.383881 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:15.383673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:51619)
0926 13:17:15.383792 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:15.383831 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260926 13:17:15.424660 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:15.424437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:44213)
0926 13:17:15.424584 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:15.424620 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260926 13:17:15.549547 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:15.549349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:58387)
0926 13:17:15.549469 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:15.549504 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260926 13:17:16.384578 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:16.384349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:42983)
0926 13:17:16.384489 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:16.384532 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260926 13:17:16.425297 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:16.425102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:44739)
0926 13:17:16.425216 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:16.425250 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260926 13:17:16.550158 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:16.549958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:59079)
0926 13:17:16.550084 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:16.550117 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260926 13:17:17.385231 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:17.385029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:44965)
0926 13:17:17.385153 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:17.385185 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260926 13:17:17.425978 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:17.425757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:59595)
0926 13:17:17.425894 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:17.425935 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 13:17:17.550859 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:17.550634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:57857)
0926 13:17:17.550777 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:17.550816 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260926 13:17:18.385991 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:18.385724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:44875)
0926 13:17:18.385883 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:18.385924 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260926 13:17:18.426718 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:18.426496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:34945)
0926 13:17:18.426637 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:18.426668 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260926 13:17:18.551456 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:18.551264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:42885)
0926 13:17:18.551385 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:18.551415 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260926 13:17:19.386654 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:19.386435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:33845)
0926 13:17:19.386569 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:19.386606 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260926 13:17:19.427363 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:19.427153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:60147)
0926 13:17:19.427281 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:19.427320 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260926 13:17:19.552050 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:19.551840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:46945)
0926 13:17:19.551975 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:19.552009 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260926 13:17:20.387363 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:20.387153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:53599)
0926 13:17:20.387278 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:20.387314 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
I20260926 13:17:20.428098 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:20.427882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:56895)
0926 13:17:20.428019 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:20.428054 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260926 13:17:20.552726 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:20.552532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:58517)
0926 13:17:20.552650 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:20.552685 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 13:17:21.387979 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:21.387779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:45339)
0926 13:17:21.387896 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:21.387932 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260926 13:17:21.428725 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:21.428549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:51097)
0926 13:17:21.428651 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:21.428682 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
I20260926 13:17:21.553340 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:21.553124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:57905)
0926 13:17:21.553251 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:21.553284 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260926 13:17:22.388967 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:22.388480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:37219)
0926 13:17:22.388847 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:22.388896 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260926 13:17:22.429481 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:22.429277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:41627)
0926 13:17:22.429393 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:22.429433 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260926 13:17:22.553993 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:22.553781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:41911)
0926 13:17:22.553908 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:22.553945 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 13:17:23.389746 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:23.389523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:50867)
0926 13:17:23.389658 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:23.389698 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260926 13:17:23.430241 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:23.429953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:44565)
0926 13:17:23.430089 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:23.430130 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260926 13:17:23.554733 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:23.554512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:51287)
0926 13:17:23.554649 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:23.554690 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260926 13:17:24.390604 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:24.390327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:38159)
0926 13:17:24.390490 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:24.390538 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260926 13:17:24.431032 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:24.430775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:52005)
0926 13:17:24.430941 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:24.430982 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260926 13:17:24.555509 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:24.555203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:48593)
0926 13:17:24.555380 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:24.555432 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260926 13:17:25.391268 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:25.391042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:51207)
0926 13:17:25.391183 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:25.391219 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260926 13:17:25.431725 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:25.431516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:38043)
0926 13:17:25.431640 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:25.431677 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260926 13:17:25.556166 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:25.555949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:48977)
0926 13:17:25.556090 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:25.556120 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260926 13:17:26.392120 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:26.391867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.1:52833)
0926 13:17:26.391994 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:26.392059 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260926 13:17:26.432449 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:26.432239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.2:38599)
0926 13:17:26.432368 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:26.432402 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260926 13:17:26.556150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15271
I20260926 13:17:26.557324 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:26.557104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43329 (local address 127.0.98.3:51637)
0926 13:17:26.557225 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:26.557265 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260926 13:17:26.560762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15432
I20260926 13:17:26.564997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15564
2026-09-26T13:17:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18509 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:17:26 dist-test-slave-f864 krb5kdc[15744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:17:26 dist-test-slave-f864 krb5kdc[15744](info): set up 2 sockets
Sep 26 13:17:26 dist-test-slave-f864 krb5kdc[15744](info): commencing operation
krb5kdc: starting...
W20260926 13:17:27.618556 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:17:27 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:17:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:17:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:17:27.736817 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:27.818655 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:27.818869 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:27.818892 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:27.820056 15757 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:17:27.820107 15757 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:17:27.820120 15757 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:17:27.820132 15757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:27.820144 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:27.820158 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:27.820169 15757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:27.820181 15757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:27.821658 15757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:27.821918 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:27.822146 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:17:27.827311 15757 server_base.cc:1034] running on GCE node
W20260926 13:17:27.827371 15763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:27.827294 15765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:27.827277 15762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:27.827769 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:27.828114 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:27.829289 15757 hybrid_clock.cc:648] HybridClock initialized: now 1790428647829273 us; error 44 us; skew 500 ppm
Sep 26 13:17:27 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:27.836750 15757 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:27.837263 15757 webserver.cc:460] Webserver started at http://127.0.98.62:35801/ using document root <none> and password file <none>
I20260926 13:17:27.837487 15757 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:27.837554 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:27.837704 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:27.838891 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "ba7d7675d5bf48cca8c9c786fd9aa395"
format_stamp: "Formatted at 2026-09-26 13:17:27 on dist-test-slave-f864"
server_key: "c49037d54c5930fee495b5d2090d20ff"
server_key_iv: "d40ef71a1fd24cbc7520383aaa65cc1b"
server_key_version: "encryptionkey@0"
I20260926 13:17:27.839263 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba7d7675d5bf48cca8c9c786fd9aa395"
format_stamp: "Formatted at 2026-09-26 13:17:27 on dist-test-slave-f864"
server_key: "c49037d54c5930fee495b5d2090d20ff"
server_key_iv: "d40ef71a1fd24cbc7520383aaa65cc1b"
server_key_version: "encryptionkey@0"
I20260926 13:17:27.840581 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:17:27.841317 15772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:27.841614 15757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:27.841727 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "ba7d7675d5bf48cca8c9c786fd9aa395"
format_stamp: "Formatted at 2026-09-26 13:17:27 on dist-test-slave-f864"
server_key: "c49037d54c5930fee495b5d2090d20ff"
server_key_iv: "d40ef71a1fd24cbc7520383aaa65cc1b"
server_key_version: "encryptionkey@0"
I20260926 13:17:27.841804 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:27.856441 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:27.858428 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:27.858596 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:27.862612 15757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45983
I20260926 13:17:27.862645 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45983 every 8 connection(s)
I20260926 13:17:27.863003 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:27.863924 15825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:27.866341 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap starting.
I20260926 13:17:27.866955 15825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:27.867318 15825 log.cc:824] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:27.867971 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: No bootstrap required, opened a new log
I20260926 13:17:27.869498 15825 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:27.869648 15825 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:27.869671 15825 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7d7675d5bf48cca8c9c786fd9aa395, State: Initialized, Role: FOLLOWER
I20260926 13:17:27.869772 15825 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:27.869853 15825 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:27.869891 15825 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:27.869942 15825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:27.870482 15825 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:27.870609 15825 leader_election.cc:302] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7d7675d5bf48cca8c9c786fd9aa395; no voters:
I20260926 13:17:27.870795 15825 leader_election.cc:288] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:17:27.870842 15828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:27.871014 15828 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 LEADER]: Becoming Leader. State: Replica: ba7d7675d5bf48cca8c9c786fd9aa395, State: Running, Role: LEADER
I20260926 13:17:27.871371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15757
I20260926 13:17:27.871477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:17:27.871311 15828 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:27.871701 392 external_mini_cluster.cc:1428] Setting key eeba1dff66731ad4cebf9ff823270ad5
I20260926 13:17:27.871061 15825 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:27.872148 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } } }
I20260926 13:17:27.872246 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:27.872188 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7d7675d5bf48cca8c9c786fd9aa395. Latest consensus state: current_term: 1 leader_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } } }
I20260926 13:17:27.872328 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:27.872489 15833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:27.873051 15833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:27.874542 15833 catalog_manager.cc:1347] Generated new cluster ID: 5a1fccd1035541a88df83e369a5424e2
I20260926 13:17:27.874595 15833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:17:27 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:27.880983 15833 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:17:27.881735 15833 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:27.883831 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:27.872292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.98.62:45983)
0926 13:17:27.872682 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:27.872695 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:27.872738 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:17:27.872856 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:27.872864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:27.872926 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:27.873095 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:27.874046 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:27.875456 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:27.876506 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:27.877159 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:27.877277 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:27.880068 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:27.880072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:27.880073 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:27.880084 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:27.882223 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:27.882622 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:27.882625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:27.882625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:27.882722 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:27.882934 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:27.882935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:27.882935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:27.883194 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:27.883324 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:27.883486 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 13:17:27.883624 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":65,"threads_started":1}
I20260926 13:17:27.886687 15833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Generated new TSK 0
I20260926 13:17:27.886854 15833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:17:27.917668 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45983
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:28.009207 15851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:28.009415 15851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:28.009438 15851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:28.010666 15851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:28.010706 15851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:28.010733 15851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:17:28.012166 15851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:28.012391 15851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:28.012650 15851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:28.017552 15857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:28.017578 15859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:28.017709 15856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:28.017882 15851 server_base.cc:1034] running on GCE node
I20260926 13:17:28.018080 15851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:28.018368 15851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:28.019512 15851 hybrid_clock.cc:648] HybridClock initialized: now 1790428648019501 us; error 29 us; skew 500 ppm
Sep 26 13:17:28 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:28.022101 15851 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:17:28.022562 15851 webserver.cc:460] Webserver started at http://127.0.98.1:37927/ using document root <none> and password file <none>
I20260926 13:17:28.022796 15851 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:28.022866 15851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:28.023012 15851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:28.024130 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "53320c4025df4910a073fa58d822917c"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "002fa341ff7d5cd6fc5ad0c11c157ef7"
server_key_iv: "e93160ede48356596db0139d64ede1df"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.024495 15851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53320c4025df4910a073fa58d822917c"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "002fa341ff7d5cd6fc5ad0c11c157ef7"
server_key_iv: "e93160ede48356596db0139d64ede1df"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.025765 15851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:17:28.026525 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.026697 15851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.026799 15851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "53320c4025df4910a073fa58d822917c"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "002fa341ff7d5cd6fc5ad0c11c157ef7"
server_key_iv: "e93160ede48356596db0139d64ede1df"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.026870 15851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:28.037202 15851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:28.039103 15851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:28.039256 15851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:28.039485 15851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:28.039778 15851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:28.039834 15851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.039875 15851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:28.039903 15851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.045723 15851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36485
I20260926 13:17:28.045782 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36485 every 8 connection(s)
I20260926 13:17:28.046065 15851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 13:17:28 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:28.053201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15851
I20260926 13:17:28.053314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:17:28.053555 392 external_mini_cluster.cc:1428] Setting key 2a05896bd55776fcd670faeb363f54dd
I20260926 13:17:28.055711 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.046801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41131 (local address 127.0.98.62:45983)
0926 13:17:28.046843 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.046845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.047051 (+ 206us) server_negotiation.cc:407] Connection header received
0926 13:17:28.047592 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.047594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.047606 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.047670 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:28.048587 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.049104 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.050115 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.050400 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.050497 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.052614 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:28.052615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:28.052616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:28.052624 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:28.054326 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.054729 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.054730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.054731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.054794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.055020 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.055020 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.055021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.055169 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:28.055234 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.055471 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.055575 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:17:28.055852 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.046766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:41131)
0926 13:17:28.046931 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.046964 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.047459 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.047738 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.047743 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.048296 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:28.048485 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.048490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.049216 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.049217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.050009 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.050010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.050507 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.050508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.050628 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.051138 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:28.051147 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:28.052514 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:28.054411 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:28.054417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:28.054419 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:28.054654 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:28.054862 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:28.054863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:28.054863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:28.054958 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:28.055231 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:28.055235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:28.055346 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.055428 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.055588 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260926 13:17:28.056504 15980 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45983
I20260926 13:17:28.056617 15980 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:28.056798 15980 heartbeater.cc:511] Master 127.0.98.62:45983 requested a full tablet report, sending...
I20260926 13:17:28.057255 15789 ts_manager.cc:194] Registered new tserver with Master: 53320c4025df4910a073fa58d822917c (127.0.98.1:36485)
I20260926 13:17:28.057849 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41131
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:17:28.077031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45983
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:28.155993 15985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:28.156214 15985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:28.156250 15985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:28.157359 15985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:28.157428 15985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:28.157485 15985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:28.158981 15985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:28.159233 15985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:28.159456 15985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:28.164453 15991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:28.164451 15990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:28.164479 15993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:28.164506 15985 server_base.cc:1034] running on GCE node
I20260926 13:17:28.164873 15985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:28.165098 15985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:28.166225 15985 hybrid_clock.cc:648] HybridClock initialized: now 1790428648166195 us; error 39 us; skew 500 ppm
Sep 26 13:17:28 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:28.168752 15985 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:28.169209 15985 webserver.cc:460] Webserver started at http://127.0.98.2:35269/ using document root <none> and password file <none>
I20260926 13:17:28.169415 15985 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:28.169456 15985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:28.169627 15985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:28.170691 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb7436a4b5f549a49b84a6d67c2dc02b"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "9c7dd297bba3755f37dc93acb3f1aa3e"
server_key_iv: "e33c36a177171ee132b503f7da7c0f75"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.171053 15985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb7436a4b5f549a49b84a6d67c2dc02b"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "9c7dd297bba3755f37dc93acb3f1aa3e"
server_key_iv: "e33c36a177171ee132b503f7da7c0f75"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.172399 15985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:17:28.173174 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.173391 15985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:28.173488 15985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "eb7436a4b5f549a49b84a6d67c2dc02b"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "9c7dd297bba3755f37dc93acb3f1aa3e"
server_key_iv: "e33c36a177171ee132b503f7da7c0f75"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.173556 15985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:28.202987 15985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:28.204901 15985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:28.205077 15985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:28.205343 15985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:28.205686 15985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:28.205742 15985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.205776 15985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:28.205804 15985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.211660 15985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36391
I20260926 13:17:28.211709 16113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36391 every 8 connection(s)
I20260926 13:17:28.212047 15985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:17:28 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:28.220640 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.212751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43351 (local address 127.0.98.62:45983)
0926 13:17:28.212807 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.212809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.213025 (+ 216us) server_negotiation.cc:407] Connection header received
0926 13:17:28.213480 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.213481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.213491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.213537 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:28.214526 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.215031 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.215978 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.216144 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.216191 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.218105 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:28.218106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:28.218107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:28.218112 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:28.219341 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.219765 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.219766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.219767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.219812 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.220095 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.220095 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.220096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.220188 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:28.220238 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.220504 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.220579 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:17:28.220883 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.212730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:43351)
0926 13:17:28.212895 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.212932 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.213393 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.213598 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.213601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.214249 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:28.214448 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.214454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.215109 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.215110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.215901 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.215903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.216211 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.216212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.216306 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.216709 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:28.216717 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:28.218021 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:28.219442 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:28.219448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:28.219450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:28.219695 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:28.219893 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:28.219893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:28.219894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:28.220009 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:28.220266 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:28.220272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:28.220390 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.220483 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.220615 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260926 13:17:28.221382 16114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45983
I20260926 13:17:28.221463 16114 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:28.221660 16114 heartbeater.cc:511] Master 127.0.98.62:45983 requested a full tablet report, sending...
I20260926 13:17:28.221895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 15985
I20260926 13:17:28.222011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:17:28.222114 15789 ts_manager.cc:194] Registered new tserver with Master: eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
I20260926 13:17:28.222260 392 external_mini_cluster.cc:1428] Setting key b657f8bd91895f751df6b98699db8014
I20260926 13:17:28.222532 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43351
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:17:28.246456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45983
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:28.327365 16119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:28.327601 16119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:28.327626 16119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:28.328835 16119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:28.328877 16119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:28.328939 16119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:28.330458 16119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:28.330729 16119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:28.330964 16119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:28.335927 16124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:28.335949 16125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:28.336042 16119 server_base.cc:1034] running on GCE node
W20260926 13:17:28.335961 16127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:28.336316 16119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:28.336556 16119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:28.337694 16119 hybrid_clock.cc:648] HybridClock initialized: now 1790428648337670 us; error 43 us; skew 500 ppm
Sep 26 13:17:28 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:28.340077 16119 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:28.340444 16119 webserver.cc:460] Webserver started at http://127.0.98.3:35711/ using document root <none> and password file <none>
I20260926 13:17:28.340617 16119 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:28.340656 16119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:28.340742 16119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:17:28.341749 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "d690e2debf9e487e873665490ef0e2d9"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "c72704e826906db7861299e2d7fd6ff4"
server_key_iv: "b03b3c80857877c321bc0015627e372d"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.342094 16119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d690e2debf9e487e873665490ef0e2d9"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "c72704e826906db7861299e2d7fd6ff4"
server_key_iv: "b03b3c80857877c321bc0015627e372d"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.343367 16119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:28.344105 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.344290 16119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:17:28.344380 16119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "d690e2debf9e487e873665490ef0e2d9"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "c72704e826906db7861299e2d7fd6ff4"
server_key_iv: "b03b3c80857877c321bc0015627e372d"
server_key_version: "encryptionkey@0"
I20260926 13:17:28.344447 16119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:28.364889 16119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:28.366909 16119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:28.367067 16119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:28.367303 16119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:28.367633 16119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:28.367690 16119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.367731 16119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:28.367755 16119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:28.373831 16119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46405
I20260926 13:17:28.373874 16247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46405 every 8 connection(s)
I20260926 13:17:28.374240 16119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:17:28 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:28.381426 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16119
I20260926 13:17:28.381542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:17:28.381763 392 external_mini_cluster.cc:1428] Setting key ed0d2ec20cba479dac38b3c8fdd745de
I20260926 13:17:28.383317 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.374891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38055 (local address 127.0.98.62:45983)
0926 13:17:28.374934 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.374936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.375145 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:17:28.375614 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.375615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.375624 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.375681 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:28.376657 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.377143 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.378120 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.378403 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.378459 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.380445 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:28.380446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:28.380447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:28.380453 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:28.381894 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.382397 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.382398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.382399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.382476 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.382761 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.382763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.382763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.382903 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:28.382965 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.383193 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.383255 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:17:28.383551 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.374869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:38055)
0926 13:17:28.375024 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.375063 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.375538 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.375754 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.375758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.376389 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:28.376582 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.376587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.377245 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.377245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.378047 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.378049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.378488 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.378489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.378592 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.379033 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:28.379041 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:28.380363 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:28.381983 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:28.381991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:28.381997 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:28.382328 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:28.382564 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:28.382565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:28.382565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:28.382694 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:28.382984 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:28.382989 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:28.383098 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.383182 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.383293 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":99,"threads_started":1}
I20260926 13:17:28.384132 16248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45983
I20260926 13:17:28.384236 16248 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:28.384402 16248 heartbeater.cc:511] Master 127.0.98.62:45983 requested a full tablet report, sending...
I20260926 13:17:28.384766 15788 ts_manager.cc:194] Registered new tserver with Master: d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405)
I20260926 13:17:28.385164 15788 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38055
I20260926 13:17:28.393872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:17:28.403107 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.395523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.0.98.62:45983)
0926 13:17:28.395604 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.395606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.395714 (+ 108us) server_negotiation.cc:407] Connection header received
0926 13:17:28.395824 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.395825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.395836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.395865 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:28.396647 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.397593 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.398647 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.398881 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.398982 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.400317 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:28.400319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:28.400319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:28.400327 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:28.401596 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.402042 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.402043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.402044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.402119 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:28.402428 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:28.402430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:28.402430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:28.402609 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:28.402742 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.402890 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.402983 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 13:17:28.404744 15788 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:17:28.406579 15788 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:17:28.413259 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.409134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36391 (local address 127.0.0.1:58838)
0926 13:17:28.409433 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.409443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.409486 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.409727 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.409730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.409733 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:28.409801 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.409804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.410923 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.410924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.411948 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.411951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.412978 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.412979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.413033 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.413043 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.413129 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.413183 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260926 13:17:28.413270 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.409259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58838 (local address 127.0.98.2:36391)
0926 13:17:28.409555 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.409560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.409569 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:17:28.409615 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.409617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.409629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.409716 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:28.409896 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.410818 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.412062 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.412812 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.412876 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.412937 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.413144 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.413184 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":85,"threads_started":1}
I20260926 13:17:28.413681 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.409134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46405 (local address 127.0.0.1:50712)
0926 13:17:28.409283 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.409328 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.409394 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.410205 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.410206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.410208 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:28.410249 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.410251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.411560 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.411561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.412572 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.412575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.413434 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.413435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.413509 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.413513 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.413570 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.413606 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260926 13:17:28.413702 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.409308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.0.98.3:46405)
0926 13:17:28.410004 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.410008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.410018 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:28.410058 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.410060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.410075 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.410322 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:28.410397 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.411425 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.412660 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.413351 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.413429 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.413506 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.413580 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.413620 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":87,"threads_started":1}
I20260926 13:17:28.414040 16048 tablet_service.cc:1467] Processing CreateTablet for tablet 81ce4a7a60054cecba2c91235020ebfe (DEFAULT_TABLE table=test-table [id=094dbdf86d5f4161ba99f7cdcd3e65a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:28.414491 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.409331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36485 (local address 127.0.0.1:43990)
0926 13:17:28.409817 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.409827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.409881 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.410151 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.410152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.410154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:28.410223 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.410226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.411528 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.411529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.412433 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.412436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.414225 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.414226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.414298 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.414302 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.414341 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.414417 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":25,"thread_start_us":81,"threads_started":1}
I20260926 13:17:28.414590 16048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ce4a7a60054cecba2c91235020ebfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:28.414480 16182 tablet_service.cc:1467] Processing CreateTablet for tablet 81ce4a7a60054cecba2c91235020ebfe (DEFAULT_TABLE table=test-table [id=094dbdf86d5f4161ba99f7cdcd3e65a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:28.414845 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ce4a7a60054cecba2c91235020ebfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:28.414896 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.409558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.0.98.1:36485)
0926 13:17:28.409746 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.409751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.409914 (+ 163us) server_negotiation.cc:407] Connection header received
0926 13:17:28.409959 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.409962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.409976 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.410069 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:28.410293 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.411414 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.412525 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.413250 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.414671 (+ 1421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.414736 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.414762 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.414811 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
I20260926 13:17:28.415518 15914 tablet_service.cc:1467] Processing CreateTablet for tablet 81ce4a7a60054cecba2c91235020ebfe (DEFAULT_TABLE table=test-table [id=094dbdf86d5f4161ba99f7cdcd3e65a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:28.415834 15914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81ce4a7a60054cecba2c91235020ebfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:28.417610 16265 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Bootstrap starting.
I20260926 13:17:28.418084 16266 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Bootstrap starting.
I20260926 13:17:28.418412 16265 tablet_bootstrap.cc:654] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:28.418792 16266 tablet_bootstrap.cc:654] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:28.419019 16267 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Bootstrap starting.
I20260926 13:17:28.419287 16265 log.cc:824] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:28.419389 16266 log.cc:824] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:28.419684 16267 tablet_bootstrap.cc:654] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:28.420080 16265 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: No bootstrap required, opened a new log
I20260926 13:17:28.420167 16265 ts_tablet_manager.cc:1397] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:17:28.420184 16266 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: No bootstrap required, opened a new log
I20260926 13:17:28.420245 16266 ts_tablet_manager.cc:1397] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:28.420236 16267 log.cc:824] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:28.421301 16267 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: No bootstrap required, opened a new log
I20260926 13:17:28.421370 16267 ts_tablet_manager.cc:1397] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:28.421753 16265 raft_consensus.cc:348] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.421916 16265 raft_consensus.cc:374] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:28.421958 16265 raft_consensus.cc:729] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d690e2debf9e487e873665490ef0e2d9, State: Initialized, Role: FOLLOWER
I20260926 13:17:28.422070 16265 consensus_queue.cc:260] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.422230 16266 raft_consensus.cc:348] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.422408 16266 raft_consensus.cc:374] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:28.422401 16265 ts_tablet_manager.cc:1428] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:17:28.422420 16248 heartbeater.cc:503] Master 127.0.98.62:45983 was elected leader, sending a full tablet report...
I20260926 13:17:28.422468 16266 raft_consensus.cc:729] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7436a4b5f549a49b84a6d67c2dc02b, State: Initialized, Role: FOLLOWER
I20260926 13:17:28.422576 16266 consensus_queue.cc:260] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.422814 16266 ts_tablet_manager.cc:1428] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:17:28.422926 16114 heartbeater.cc:503] Master 127.0.98.62:45983 was elected leader, sending a full tablet report...
I20260926 13:17:28.423867 16267 raft_consensus.cc:348] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.423995 16267 raft_consensus.cc:374] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:28.424014 16267 raft_consensus.cc:729] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53320c4025df4910a073fa58d822917c, State: Initialized, Role: FOLLOWER
I20260926 13:17:28.424113 16267 consensus_queue.cc:260] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.424322 16267 ts_tablet_manager.cc:1428] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 13:17:28.424396 15980 heartbeater.cc:503] Master 127.0.98.62:45983 was elected leader, sending a full tablet report...
I20260926 13:17:28.429837 16272 raft_consensus.cc:482] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:17:28.429903 16272 raft_consensus.cc:504] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.430254 16272 leader_election.cc:288] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), 53320c4025df4910a073fa58d822917c (127.0.98.1:36485)
I20260926 13:17:28.433059 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.430495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60049 (local address 127.0.98.3:46405)
0926 13:17:28.430555 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.430556 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.430566 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:28.430619 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.430619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.430627 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.430698 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:28.430846 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.431403 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.432316 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.432712 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.432787 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.432799 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.432943 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.432995 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:17:28.433058 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.430434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46405 (local address 127.0.98.2:60049)
0926 13:17:28.430502 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.430515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.430555 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.430703 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.430704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.430706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:28.430772 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.430774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.431487 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.431488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.432232 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.432234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.432787 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.432788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.432865 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.432869 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.432920 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.432977 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":21}
I20260926 13:17:28.433099 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.430734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54421 (local address 127.0.98.1:36485)
0926 13:17:28.430780 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.430782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.430789 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:17:28.430808 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.430809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.430819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.430871 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:28.431022 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.431467 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.432378 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.432717 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.432800 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.432809 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.432998 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.433034 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:17:28.433132 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.430550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36485 (local address 127.0.98.2:54421)
0926 13:17:28.430672 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.430681 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:28.430721 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:28.430889 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:28.430889 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:28.430891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:28.430954 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.430955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.431551 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.431552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.432262 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:28.432264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.432818 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.432819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:28.432896 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.432899 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:17:28.432974 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:17:28.433009 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260926 13:17:28.433426 15934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ce4a7a60054cecba2c91235020ebfe" candidate_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53320c4025df4910a073fa58d822917c" is_pre_election: true
I20260926 13:17:28.433457 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ce4a7a60054cecba2c91235020ebfe" candidate_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d690e2debf9e487e873665490ef0e2d9" is_pre_election: true
I20260926 13:17:28.433574 16202 raft_consensus.cc:2454] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb7436a4b5f549a49b84a6d67c2dc02b in term 0.
I20260926 13:17:28.433575 15934 raft_consensus.cc:2454] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb7436a4b5f549a49b84a6d67c2dc02b in term 0.
I20260926 13:17:28.433765 16004 leader_election.cc:302] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b; no voters:
I20260926 13:17:28.433904 16272 raft_consensus.cc:2781] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:17:28.433974 16272 raft_consensus.cc:482] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:17:28.434010 16272 raft_consensus.cc:3037] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:28.434718 16272 raft_consensus.cc:504] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.434846 16272 leader_election.cc:288] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [CANDIDATE]: Term 1 election: Requested vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), 53320c4025df4910a073fa58d822917c (127.0.98.1:36485)
I20260926 13:17:28.435024 16202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ce4a7a60054cecba2c91235020ebfe" candidate_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d690e2debf9e487e873665490ef0e2d9"
I20260926 13:17:28.435031 15934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81ce4a7a60054cecba2c91235020ebfe" candidate_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53320c4025df4910a073fa58d822917c"
I20260926 13:17:28.435086 16202 raft_consensus.cc:3037] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:28.435118 15934 raft_consensus.cc:3037] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:28.435760 16202 raft_consensus.cc:2454] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7436a4b5f549a49b84a6d67c2dc02b in term 1.
I20260926 13:17:28.435858 15934 raft_consensus.cc:2454] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb7436a4b5f549a49b84a6d67c2dc02b in term 1.
I20260926 13:17:28.435916 16004 leader_election.cc:302] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b; no voters:
I20260926 13:17:28.436028 16272 raft_consensus.cc:2781] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:28.436170 16272 raft_consensus.cc:686] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 LEADER]: Becoming Leader. State: Replica: eb7436a4b5f549a49b84a6d67c2dc02b, State: Running, Role: LEADER
I20260926 13:17:28.436234 16272 consensus_queue.cc:237] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:28.437000 15788 catalog_manager.cc:5495] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } health_report { overall_health: HEALTHY } } }
I20260926 13:17:28.444029 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.441923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58846 (local address 127.0.98.2:36391)
0926 13:17:28.441969 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:28.441971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:28.441978 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:17:28.442039 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:28.442040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:28.442055 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:28.442115 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:28.442252 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.442672 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.443372 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:28.443542 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:28.443589 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:28.443710 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:28.443793 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:17:28.443845 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:28.443911 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 13:17:28.443925 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:17:28.445768 15934 raft_consensus.cc:1261] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Refusing update from remote peer eb7436a4b5f549a49b84a6d67c2dc02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:28.445768 16202 raft_consensus.cc:1261] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Refusing update from remote peer eb7436a4b5f549a49b84a6d67c2dc02b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:28.446096 16272 consensus_queue.cc:1035] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:28.446288 16275 consensus_queue.cc:1035] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:28.448900 16279 mvcc.cc:205] Tried to move back new op lower bound from 7333595744031158272 to 7333595743995449344. Current Snapshot: MvccSnapshot[applied={T|T < 7333595744031158272}]
I20260926 13:17:28.451074 15788 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:28.451172 15788 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:28.451901 15788 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Sending DeleteTablet for 3 replicas of tablet 81ce4a7a60054cecba2c91235020ebfe
I20260926 13:17:28.452207 16182 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ce4a7a60054cecba2c91235020ebfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50712
I20260926 13:17:28.452289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15851
I20260926 13:17:28.452425 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ce4a7a60054cecba2c91235020ebfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43990
I20260926 13:17:28.452860 16287 tablet_replica.cc:330] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: stopping tablet replica
I20260926 13:17:28.452955 16287 raft_consensus.cc:2229] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:28.453056 16287 raft_consensus.cc:2258] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:28.453474 16287 ts_tablet_manager.cc:1905] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:28.454888 16048 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ce4a7a60054cecba2c91235020ebfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58838
I20260926 13:17:28.455181 16288 tablet_replica.cc:330] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: stopping tablet replica
I20260926 13:17:28.455261 16288 raft_consensus.cc:2229] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:17:28.455363 16288 raft_consensus.cc:2258] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:28.455389 16287 ts_tablet_manager.cc:1918] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:28.455451 16287 log.cc:1196] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/81ce4a7a60054cecba2c91235020ebfe
I20260926 13:17:28.455720 16287 ts_tablet_manager.cc:1939] T 81ce4a7a60054cecba2c91235020ebfe P d690e2debf9e487e873665490ef0e2d9: Deleting consensus metadata
I20260926 13:17:28.455802 16288 ts_tablet_manager.cc:1905] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:28.456102 15776 catalog_manager.cc:4841] TS d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): tablet 81ce4a7a60054cecba2c91235020ebfe (table test-table [id=094dbdf86d5f4161ba99f7cdcd3e65a1]) successfully deleted
I20260926 13:17:28.457629 16288 ts_tablet_manager.cc:1918] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:28.457686 16288 log.cc:1196] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/81ce4a7a60054cecba2c91235020ebfe
I20260926 13:17:28.457926 16288 ts_tablet_manager.cc:1939] T 81ce4a7a60054cecba2c91235020ebfe P eb7436a4b5f549a49b84a6d67c2dc02b: Deleting consensus metadata
I20260926 13:17:28.458351 15775 catalog_manager.cc:4841] TS eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): tablet 81ce4a7a60054cecba2c91235020ebfe (table test-table [id=094dbdf86d5f4161ba99f7cdcd3e65a1]) successfully deleted
W20260926 13:17:28.458861 15776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36485 (error 108)
I20260926 13:17:28.459342 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:28.459214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36485 (local address 127.0.0.1:43992)
0926 13:17:28.459273 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:28.459309 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:17:28.459379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15985
W20260926 13:17:28.460855 15776 catalog_manager.cc:4584] TS 53320c4025df4910a073fa58d822917c (127.0.98.1:36485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81ce4a7a60054cecba2c91235020ebfe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36485: connect: Connection refused (error 111)
I20260926 13:17:28.464617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 16119
I20260926 13:17:28.469110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 15757
I20260926 13:17:48.474567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45983
--webserver_interface=127.0.98.62
--webserver_port=35801
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:17:48.554883 16290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:48.555083 16290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:48.555105 16290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:48.556174 16290 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:17:48.556205 16290 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:17:48.556218 16290 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:17:48.556231 16290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:17:48.556242 16290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:48.556255 16290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:48.556280 16290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:17:48.556293 16290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:17:48.557672 16290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:48.557857 16290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:48.558020 16290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:48.566702 16297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:48.566679 16296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:48.566790 16290 server_base.cc:1034] running on GCE node
W20260926 13:17:48.566679 16299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:48.567142 16290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:48.567456 16290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:48.568616 16290 hybrid_clock.cc:648] HybridClock initialized: now 1790428668568622 us; error 44 us; skew 500 ppm
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:48.571235 16290 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:17:48.571844 16290 webserver.cc:460] Webserver started at http://127.0.98.62:35801/ using document root <none> and password file <none>
I20260926 13:17:48.572144 16290 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:48.572242 16290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:48.573845 16290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:17:48.574607 16306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.574800 16290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.574908 16290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "ba7d7675d5bf48cca8c9c786fd9aa395"
format_stamp: "Formatted at 2026-09-26 13:17:27 on dist-test-slave-f864"
server_key: "c49037d54c5930fee495b5d2090d20ff"
server_key_iv: "d40ef71a1fd24cbc7520383aaa65cc1b"
server_key_version: "encryptionkey@0"
I20260926 13:17:48.575320 16290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:48.582412 16290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:48.584246 16290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:48.584396 16290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:48.588435 16290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45983
I20260926 13:17:48.588470 16358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45983 every 8 connection(s)
I20260926 13:17:48.588858 16290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:17:48.589257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16290
I20260926 13:17:48.589576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36485
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45983
--webserver_port=37927
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:48.590539 16359 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:17:48.591192 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap starting.
I20260926 13:17:48.599813 16359 log.cc:824] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:48.601687 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:48.601969 16359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap complete.
I20260926 13:17:48.604234 16359 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:48.604480 16359 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7d7675d5bf48cca8c9c786fd9aa395, State: Initialized, Role: FOLLOWER
I20260926 13:17:48.604682 16359 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:48.604782 16359 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:17:48.604841 16359 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:17:48.604887 16359 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 1 FOLLOWER]: Advancing to term 2
I20260926 13:17:48.605891 16359 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:48.606007 16359 leader_election.cc:302] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7d7675d5bf48cca8c9c786fd9aa395; no voters:
I20260926 13:17:48.606226 16359 leader_election.cc:288] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 13:17:48.606285 16363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 13:17:48.606458 16363 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 LEADER]: Becoming Leader. State: Replica: ba7d7675d5bf48cca8c9c786fd9aa395, State: Running, Role: LEADER
I20260926 13:17:48.606487 16359 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:17:48.606621 16363 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:17:48.606942 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7d7675d5bf48cca8c9c786fd9aa395. Latest consensus state: current_term: 2 leader_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } } }
I20260926 13:17:48.607028 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:48.606957 16364 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } } }
I20260926 13:17:48.607079 16364 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: This master's current role is: LEADER
I20260926 13:17:48.607709 16371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:17:48.608563 16371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:17:48.608687 16371 catalog_manager.cc:1259] Loaded cluster ID: 5a1fccd1035541a88df83e369a5424e2
I20260926 13:17:48.608729 16371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:17:48.610046 16371 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:17:48.610288 16371 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Loaded TSK: 0
I20260926 13:17:48.623844 16371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Generated new TSK 1
I20260926 13:17:48.623962 16371 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:17:48.679479 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:48.679693 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:48.679728 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:48.680882 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:48.680926 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:48.680953 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:17:48.682431 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37927
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:48.682660 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:48.682896 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:48.688503 16386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:48.688511 16389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:48.688583 16387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:48.689069 16361 server_base.cc:1034] running on GCE node
I20260926 13:17:48.689260 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:48.689498 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:48.690654 16361 hybrid_clock.cc:648] HybridClock initialized: now 1790428668690641 us; error 31 us; skew 500 ppm
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:48.693421 16361 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:17:48.693877 16361 webserver.cc:460] Webserver started at http://127.0.98.1:37927/ using document root <none> and password file <none>
I20260926 13:17:48.694146 16361 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:48.694265 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:48.695757 16361 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:48.696465 16396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.696672 16361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:48.696767 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "53320c4025df4910a073fa58d822917c"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "002fa341ff7d5cd6fc5ad0c11c157ef7"
server_key_iv: "e93160ede48356596db0139d64ede1df"
server_key_version: "encryptionkey@0"
I20260926 13:17:48.697145 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:48.710937 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:48.713136 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:48.713301 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:48.713522 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:48.713982 16403 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:17:48.715896 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:17:48.715955 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:17:48.715996 16361 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:17:48.716611 16361 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:17:48.716666 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:17:48.716744 16403 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Bootstrap starting.
I20260926 13:17:48.723766 16361 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36485
I20260926 13:17:48.723837 16510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36485 every 8 connection(s)
I20260926 13:17:48.724188 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:17:48.724494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16361
I20260926 13:17:48.724793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36391
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45983
--webserver_port=35269
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:48.728008 16403 log.cc:824] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:48.730469 16403 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:17:48.730775 16403 tablet_bootstrap.cc:492] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Bootstrap complete.
I20260926 13:17:48.731940 16403 ts_tablet_manager.cc:1397] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260926 13:17:48.733448 16403 raft_consensus.cc:348] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:48.733743 16403 raft_consensus.cc:729] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53320c4025df4910a073fa58d822917c, State: Initialized, Role: FOLLOWER
I20260926 13:17:48.734043 16403 consensus_queue.cc:260] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } }
I20260926 13:17:48.734467 16403 ts_tablet_manager.cc:1428] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:48.741629 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.726821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:39339)
0926 13:17:48.728238 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:48.728296 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:48.729016 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:48.729700 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:48.729705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:48.730108 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:48.730450 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.730457 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.732308 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.732310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.733726 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.733729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.734777 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.734778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.734929 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.735617 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:48.735628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:48.737497 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:48.739763 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.739770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.739772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.740059 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:48.740393 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.740394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.740394 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.740521 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:48.740978 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:48.740983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:48.741139 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 13:17:48.741249 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:17:48.741424 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1369,"thread_start_us":111,"threads_started":1}
I20260926 13:17:48.741636 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.726955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39339 (local address 127.0.98.62:45983)
0926 13:17:48.727132 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:48.727140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:48.729181 (+ 2041us) server_negotiation.cc:407] Connection header received
0926 13:17:48.729347 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:48.729354 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:48.729413 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:48.729546 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:48.730597 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.732167 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.733832 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.734562 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.734673 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.737611 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:48.737615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:48.737616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:48.737628 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:48.739665 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.740147 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.740151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.740151 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.740302 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.740598 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.740599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.740599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.740899 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:48.740989 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:48.741330 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 13:17:48.741468 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260926 13:17:48.742904 16511 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45983
I20260926 13:17:48.743005 16511 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:48.743176 16511 heartbeater.cc:511] Master 127.0.98.62:45983 requested a full tablet report, sending...
I20260926 13:17:48.743942 16323 ts_manager.cc:194] Registered new tserver with Master: 53320c4025df4910a073fa58d822917c (127.0.98.1:36485)
I20260926 13:17:48.745074 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39339
I20260926 13:17:48.746054 16511 heartbeater.cc:503] Master 127.0.98.62:45983 was elected leader, sending a full tablet report...
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 13:17:48.752638 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.744651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.0.98.1:36485)
0926 13:17:48.744806 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:48.744810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:48.744855 (+ 45us) server_negotiation.cc:407] Connection header received
0926 13:17:48.744964 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:48.744966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:48.744980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:48.745058 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:48.745549 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.746294 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.747500 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.747789 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.747844 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.749865 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:48.749872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:48.749873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:48.749879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:48.751377 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.751784 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.751787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.751787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.751847 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.752051 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.752052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.752052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.752190 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:48.752285 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:48.752460 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 13:17:48.752548 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260926 13:17:48.752638 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.744568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36485 (local address 127.0.0.1:40284)
0926 13:17:48.744774 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:48.744804 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:48.744871 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:48.745077 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:48.745081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:48.745409 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:48.745464 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.745466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.746498 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.746499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.747418 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.747421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.747860 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.747863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.747938 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.748379 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:48.748384 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:48.749761 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:48.751476 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.751482 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.751484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.751680 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:48.751923 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.751924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.751924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.751987 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:48.752264 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:48.752271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:48.752322 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 13:17:48.752424 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 13:17:48.752542 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
I20260926 13:17:48.753067 16445 tablet_service.cc:1514] Processing DeleteTablet for tablet 81ce4a7a60054cecba2c91235020ebfe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40284
I20260926 13:17:48.753579 16520 tablet_replica.cc:330] stopping tablet replica
I20260926 13:17:48.753681 16520 raft_consensus.cc:2229] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:48.753750 16520 raft_consensus.cc:2258] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:48.754045 16520 ts_tablet_manager.cc:1905] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:48.756237 16520 ts_tablet_manager.cc:1918] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:48.756296 16520 log.cc:1196] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal/wals/81ce4a7a60054cecba2c91235020ebfe
I20260926 13:17:48.756531 16520 ts_tablet_manager.cc:1939] T 81ce4a7a60054cecba2c91235020ebfe P 53320c4025df4910a073fa58d822917c: Deleting consensus metadata
I20260926 13:17:48.756994 16310 catalog_manager.cc:4841] TS 53320c4025df4910a073fa58d822917c (127.0.98.1:36485): tablet 81ce4a7a60054cecba2c91235020ebfe (table test-table [id=094dbdf86d5f4161ba99f7cdcd3e65a1]) successfully deleted
W20260926 13:17:48.816202 16513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:48.816458 16513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:48.816505 16513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:48.817780 16513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:48.817854 16513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:48.817907 16513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:17:48.819670 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35269
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:48.819977 16513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:48.820263 16513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:48.825924 16526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:48.825914 16525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:48.825958 16513 server_base.cc:1034] running on GCE node
W20260926 13:17:48.825927 16528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:48.826314 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:48.826570 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:48.827728 16513 hybrid_clock.cc:648] HybridClock initialized: now 1790428668827706 us; error 40 us; skew 500 ppm
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:48.830250 16513 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:17:48.830621 16513 webserver.cc:460] Webserver started at http://127.0.98.2:35269/ using document root <none> and password file <none>
I20260926 13:17:48.830818 16513 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:48.830871 16513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:48.832115 16513 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:48.832648 16535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.832845 16513 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.832948 16513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "eb7436a4b5f549a49b84a6d67c2dc02b"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "9c7dd297bba3755f37dc93acb3f1aa3e"
server_key_iv: "e33c36a177171ee132b503f7da7c0f75"
server_key_version: "encryptionkey@0"
I20260926 13:17:48.833272 16513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:48.854456 16513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:48.856823 16513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:48.856936 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:48.857118 16513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:48.857438 16513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:48.857473 16513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.857497 16513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:48.857547 16513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.864643 16513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36391
I20260926 13:17:48.864727 16648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36391 every 8 connection(s)
I20260926 13:17:48.865073 16513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:48.872535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16513
I20260926 13:17:48.872848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46405
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45983
--webserver_port=35711
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:17:48.874637 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.865826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52491 (local address 127.0.98.62:45983)
0926 13:17:48.865885 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:48.865887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:48.866130 (+ 243us) server_negotiation.cc:407] Connection header received
0926 13:17:48.866660 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:48.866661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:48.866679 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:48.866737 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:48.867703 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.868219 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.869193 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.869414 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.869478 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.871704 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:48.871705 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:48.871705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:48.871712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:48.873242 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.873616 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.873617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.873618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.873678 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.873922 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.873922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.873923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.874070 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:48.874142 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:48.874498 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 13:17:48.874573 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:17:48.874912 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.865823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:52491)
0926 13:17:48.865976 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:48.866027 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:48.866568 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:48.866805 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:48.866809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:48.867410 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:48.867603 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.867607 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.868302 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.868303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.869102 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.869104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.869496 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.869496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.869600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.870067 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:48.870075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:48.871612 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:48.873322 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.873328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.873330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.873542 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:48.873755 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.873756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.873756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.873858 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:48.874142 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:48.874149 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:48.874353 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 13:17:48.874477 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 13:17:48.874654 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260926 13:17:48.875487 16649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45983
I20260926 13:17:48.875571 16649 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:48.875758 16649 heartbeater.cc:511] Master 127.0.98.62:45983 requested a full tablet report, sending...
I20260926 13:17:48.876157 16323 ts_manager.cc:194] Registered new tserver with Master: eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
I20260926 13:17:48.876526 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52491
W20260926 13:17:48.953889 16652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:17:48.954130 16652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:17:48.954167 16652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:17:48.955328 16652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:17:48.955403 16652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:17:48.955451 16652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:17:48.956887 16652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35711
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:17:48.957160 16652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:17:48.957384 16652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:17:48.962369 16657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:48.962379 16660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:17:48.962379 16658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:17:48.962551 16652 server_base.cc:1034] running on GCE node
I20260926 13:17:48.962780 16652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:17:48.963016 16652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:17:48.964181 16652 hybrid_clock.cc:648] HybridClock initialized: now 1790428668964167 us; error 32 us; skew 500 ppm
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:17:48.966885 16652 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:17:48.967342 16652 webserver.cc:460] Webserver started at http://127.0.98.3:35711/ using document root <none> and password file <none>
I20260926 13:17:48.967576 16652 fs_manager.cc:359] Metadata directory not provided
I20260926 13:17:48.967631 16652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:17:48.968906 16652 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:48.969550 16667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.969776 16652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:17:48.969873 16652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "d690e2debf9e487e873665490ef0e2d9"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "c72704e826906db7861299e2d7fd6ff4"
server_key_iv: "b03b3c80857877c321bc0015627e372d"
server_key_version: "encryptionkey@0"
I20260926 13:17:48.970276 16652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:17:48.979067 16652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:17:48.981328 16652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:17:48.981482 16652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:17:48.981705 16652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:17:48.981994 16652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:17:48.982048 16652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.982088 16652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:17:48.982116 16652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:48.987965 16652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46405
I20260926 13:17:48.988014 16780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46405 every 8 connection(s)
I20260926 13:17:48.988333 16652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:17:48.988516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16652
Sep 26 13:17:48 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:17:48.998142 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.990717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.62:45983)
0926 13:17:48.990778 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:48.990780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:48.990791 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:48.991007 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:48.991007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:48.991018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:48.991069 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:48.991847 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.992436 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.993488 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.993778 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.993855 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.995307 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:48.995309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:48.995309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:48.995317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:48.996940 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.997308 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.997309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.997310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.997375 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:48.997559 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:48.997560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:48.997560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:48.997701 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:48.997893 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:48.997935 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:17:48.998015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 13:17:49.001001 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.990831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41567 (local address 127.0.98.62:45983)
0926 13:17:48.990962 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:48.990964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:48.991315 (+ 351us) server_negotiation.cc:407] Connection header received
0926 13:17:48.992008 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:48.992009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:48.992022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:48.992111 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:48.993223 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.993862 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.995455 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.995721 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.995797 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.998044 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:48.998045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:48.998045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:48.998053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:48.999651 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:49.000059 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:49.000060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:49.000061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:49.000127 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:49.000338 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:49.000339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:49.000339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:49.000433 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:49.000517 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.000862 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.000940 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260926 13:17:49.001152 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:48.990868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:41567)
0926 13:17:48.991172 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:48.991219 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:48.991925 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:48.992165 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:48.992168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:48.992869 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:17:48.993129 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.993136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.993953 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.993954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.995383 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:48.995386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:48.995817 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:48.995818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:48.995948 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:48.996452 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:17:48.996460 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:17:48.997948 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:17:48.999733 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:48.999739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:48.999742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:48.999961 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:49.000187 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:17:49.000188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:17:49.000188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:17:49.000281 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:17:49.000526 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:17:49.000533 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:17:49.000711 (+ 178us) client_negotiation.cc:769] Sending connection context
0926 13:17:49.000824 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 13:17:49.000943 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260926 13:17:49.001631 16781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45983
I20260926 13:17:49.001729 16781 heartbeater.cc:464] Registering TS with master...
I20260926 13:17:49.001907 16781 heartbeater.cc:511] Master 127.0.98.62:45983 requested a full tablet report, sending...
I20260926 13:17:49.002300 16323 ts_manager.cc:194] Registered new tserver with Master: d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405)
I20260926 13:17:49.002660 16323 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41567
I20260926 13:17:49.009737 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 13:17:49.013485 16251 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:45983)
I20260926 13:17:49.013296 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.010136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.62:45983)
0926 13:17:49.010204 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.010206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.010389 (+ 183us) server_negotiation.cc:407] Connection header received
0926 13:17:49.010494 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.010497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.010535 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.010613 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:49.010736 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.011204 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.012310 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.012572 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.012669 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.012866 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:49.012905 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 13:17:49.013219 (+ 314us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57536: token expired
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 13:17:49.014142 16784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57536: token expired
I20260926 13:17:49.021241 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.013782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57542 (local address 127.0.98.62:45983)
0926 13:17:49.013846 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.013848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.013857 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:17:49.013950 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.013951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.013960 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.013987 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:17:49.014882 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.015336 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.016469 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.016838 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.016926 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.018387 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:17:49.018389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:17:49.018389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:17:49.018409 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:17:49.020123 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:49.020486 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:49.020486 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:49.020487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:49.020546 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:17:49.020749 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:17:49.020750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:17:49.020750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:17:49.020910 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:17:49.020996 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.021115 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.021181 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:17:49.022315 16323 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:17:49.023917 16323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:17:49.026492 16445 tablet_service.cc:1467] Processing CreateTablet for tablet f25abd4284db49158ab9637ea76ad485 (DEFAULT_TABLE table=test-table [id=34eec282e3844372b1fd845e8447374a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:49.026728 16445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25abd4284db49158ab9637ea76ad485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:49.028404 16403 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Bootstrap starting.
I20260926 13:17:49.028716 16403 tablet_bootstrap.cc:654] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:49.029448 16403 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: No bootstrap required, opened a new log
I20260926 13:17:49.029517 16403 ts_tablet_manager.cc:1397] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 13:17:49.029681 16403 raft_consensus.cc:348] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.029765 16403 raft_consensus.cc:374] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:49.029817 16403 raft_consensus.cc:729] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53320c4025df4910a073fa58d822917c, State: Initialized, Role: FOLLOWER
I20260926 13:17:49.029870 16403 consensus_queue.cc:260] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.029968 16403 ts_tablet_manager.cc:1428] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 13:17:49.030386 16787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.025952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46405 (local address 127.0.0.1:36420)
0926 13:17:49.026126 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:49.026135 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:49.026228 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:49.026567 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:49.026568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:49.026571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:49.026628 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.026630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.027725 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.027726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.028840 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.028842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.030069 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.030070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.030144 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.030196 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 13:17:49.030265 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:17:49.030309 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":21,"thread_start_us":94,"threads_started":1}
I20260926 13:17:49.030501 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.025950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36391 (local address 127.0.0.1:57032)
0926 13:17:49.026067 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:49.026081 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:49.026129 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:49.026722 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:49.026724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:49.026725 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:49.026771 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.026773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.028144 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.028145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.029182 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.029184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.030238 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.030239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.030321 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.030325 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:17:49.030382 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:17:49.030434 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260926 13:17:49.030531 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.026303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.0.98.2:36391)
0926 13:17:49.026495 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.026500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.026511 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:17:49.026559 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.026561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.026577 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.026906 (+ 329us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:49.027007 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.028049 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.029267 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.030110 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.030230 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.030291 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.030410 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.030454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":98,"threads_started":1}
I20260926 13:17:49.030406 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.026074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.3:46405)
0926 13:17:49.026377 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.026382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.026392 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:49.026436 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.026438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.026454 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.026542 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:49.026807 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.027626 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.029163 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.029955 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.030061 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.030134 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.030282 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.030328 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":76,"threads_started":1}
I20260926 13:17:49.031267 16583 tablet_service.cc:1467] Processing CreateTablet for tablet f25abd4284db49158ab9637ea76ad485 (DEFAULT_TABLE table=test-table [id=34eec282e3844372b1fd845e8447374a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:49.031421 16715 tablet_service.cc:1467] Processing CreateTablet for tablet f25abd4284db49158ab9637ea76ad485 (DEFAULT_TABLE table=test-table [id=34eec282e3844372b1fd845e8447374a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:17:49.031625 16583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25abd4284db49158ab9637ea76ad485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:49.031767 16715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f25abd4284db49158ab9637ea76ad485. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:17:49.034490 16791 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Bootstrap starting.
I20260926 13:17:49.035069 16791 tablet_bootstrap.cc:654] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:49.035573 16791 log.cc:824] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:49.035897 16792 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Bootstrap starting.
I20260926 13:17:49.036348 16791 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: No bootstrap required, opened a new log
I20260926 13:17:49.036430 16791 ts_tablet_manager.cc:1397] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:17:49.036506 16792 tablet_bootstrap.cc:654] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Neither blocks nor log segments found. Creating new log.
I20260926 13:17:49.037015 16792 log.cc:824] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:17:49.037875 16792 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: No bootstrap required, opened a new log
I20260926 13:17:49.037949 16792 ts_tablet_manager.cc:1397] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:17:49.038106 16791 raft_consensus.cc:348] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.038344 16791 raft_consensus.cc:374] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:49.038393 16791 raft_consensus.cc:729] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb7436a4b5f549a49b84a6d67c2dc02b, State: Initialized, Role: FOLLOWER
I20260926 13:17:49.038516 16791 consensus_queue.cc:260] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.038780 16791 ts_tablet_manager.cc:1428] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:17:49.038802 16649 heartbeater.cc:503] Master 127.0.98.62:45983 was elected leader, sending a full tablet report...
I20260926 13:17:49.039608 16792 raft_consensus.cc:348] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.039746 16792 raft_consensus.cc:374] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:17:49.039769 16792 raft_consensus.cc:729] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d690e2debf9e487e873665490ef0e2d9, State: Initialized, Role: FOLLOWER
I20260926 13:17:49.039844 16792 consensus_queue.cc:260] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.040071 16792 ts_tablet_manager.cc:1428] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:17:49.040105 16781 heartbeater.cc:503] Master 127.0.98.62:45983 was elected leader, sending a full tablet report...
W20260926 13:17:49.115638 16650 tablet.cc:2369] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:17:49.226732 16512 tablet.cc:2369] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:17:49.240787 16782 tablet.cc:2369] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:17:49.304019 16796 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:17:49.304148 16796 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.304593 16796 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391), 53320c4025df4910a073fa58d822917c (127.0.98.1:36485)
I20260926 13:17:49.309275 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.304888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58067 (local address 127.0.98.2:36391)
0926 13:17:49.304937 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.304939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.305163 (+ 224us) server_negotiation.cc:407] Connection header received
0926 13:17:49.305226 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.305228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.305241 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.305296 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:49.305551 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.306324 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.308212 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.308847 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.308930 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.308942 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.309164 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.309207 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:17:49.309306 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.304938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39749 (local address 127.0.98.1:36485)
0926 13:17:49.305075 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.305078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.305091 (+ 13us) server_negotiation.cc:407] Connection header received
0926 13:17:49.305153 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.305154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.305188 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.305282 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:17:49.305552 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.306370 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.308216 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.308859 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.308930 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.309030 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.309164 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.309214 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260926 13:17:49.309289 16785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.304818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36485 (local address 127.0.98.3:39749)
0926 13:17:49.304954 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:49.304968 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:49.305031 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:49.305327 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:49.305328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:49.305330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:49.305456 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.305458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.306439 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.306440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.308093 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.308094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.308972 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.308973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.309067 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.309073 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:17:49.309143 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 13:17:49.309217 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":130}
I20260926 13:17:49.309297 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.304878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36391 (local address 127.0.98.3:58067)
0926 13:17:49.305079 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:49.305090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:17:49.305153 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:17:49.305327 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:17:49.305328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:17:49.305330 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:17:49.305464 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.305466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.306435 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.306436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.308085 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:17:49.308088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.308976 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.308976 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:17:49.309067 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.309074 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:17:49.309143 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:17:49.309191 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":83,"thread_start_us":114,"threads_started":1}
I20260926 13:17:49.309639 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25abd4284db49158ab9637ea76ad485" candidate_uuid: "d690e2debf9e487e873665490ef0e2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53320c4025df4910a073fa58d822917c" is_pre_election: true
I20260926 13:17:49.309829 16465 raft_consensus.cc:2454] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d690e2debf9e487e873665490ef0e2d9 in term 0.
I20260926 13:17:49.309923 16603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25abd4284db49158ab9637ea76ad485" candidate_uuid: "d690e2debf9e487e873665490ef0e2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" is_pre_election: true
I20260926 13:17:49.310091 16603 raft_consensus.cc:2454] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d690e2debf9e487e873665490ef0e2d9 in term 0.
I20260926 13:17:49.310146 16671 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53320c4025df4910a073fa58d822917c, d690e2debf9e487e873665490ef0e2d9; no voters:
I20260926 13:17:49.310357 16796 raft_consensus.cc:2781] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:17:49.310437 16796 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:17:49.310477 16796 raft_consensus.cc:3037] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:49.312567 16796 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.312724 16796 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [CANDIDATE]: Term 1 election: Requested vote from peers eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391), 53320c4025df4910a073fa58d822917c (127.0.98.1:36485)
I20260926 13:17:49.312880 16603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25abd4284db49158ab9637ea76ad485" candidate_uuid: "d690e2debf9e487e873665490ef0e2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b"
I20260926 13:17:49.312897 16465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25abd4284db49158ab9637ea76ad485" candidate_uuid: "d690e2debf9e487e873665490ef0e2d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53320c4025df4910a073fa58d822917c"
I20260926 13:17:49.312968 16465 raft_consensus.cc:3037] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:49.312968 16603 raft_consensus.cc:3037] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:17:49.313772 16465 raft_consensus.cc:2454] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d690e2debf9e487e873665490ef0e2d9 in term 1.
I20260926 13:17:49.313787 16603 raft_consensus.cc:2454] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d690e2debf9e487e873665490ef0e2d9 in term 1.
I20260926 13:17:49.314003 16671 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53320c4025df4910a073fa58d822917c, d690e2debf9e487e873665490ef0e2d9; no voters:
I20260926 13:17:49.314098 16796 raft_consensus.cc:2781] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:17:49.314281 16796 raft_consensus.cc:686] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 1 LEADER]: Becoming Leader. State: Replica: d690e2debf9e487e873665490ef0e2d9, State: Running, Role: LEADER
I20260926 13:17:49.314384 16796 consensus_queue.cc:237] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:17:49.315200 16323 catalog_manager.cc:5495] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d690e2debf9e487e873665490ef0e2d9 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d690e2debf9e487e873665490ef0e2d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } health_report { overall_health: UNKNOWN } } }
I20260926 13:17:49.381798 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.379368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.0.98.3:46405)
0926 13:17:49.379438 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 13:17:49.379440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:17:49.379450 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:17:49.379494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:17:49.379495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:17:49.379526 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:17:49.379604 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:17:49.379728 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.380208 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.380974 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:17:49.381180 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:17:49.381271 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:17:49.381436 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:17:49.381544 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:17:49.381632 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:17:49.381670 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 13:17:49.381690 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:17:49.383708 16465 raft_consensus.cc:1261] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Refusing update from remote peer d690e2debf9e487e873665490ef0e2d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:49.383700 16603 raft_consensus.cc:1261] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Refusing update from remote peer d690e2debf9e487e873665490ef0e2d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:17:49.384110 16799 consensus_queue.cc:1035] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:49.384258 16796 consensus_queue.cc:1035] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:17:49.386813 16804 mvcc.cc:205] Tried to move back new op lower bound from 7333595829792813056 to 7333595829512241152. Current Snapshot: MvccSnapshot[applied={T|T < 7333595829792813056}]
I20260926 13:17:49.387073 16803 mvcc.cc:205] Tried to move back new op lower bound from 7333595829792813056 to 7333595829512241152. Current Snapshot: MvccSnapshot[applied={T|T < 7333595829792813056}]
I20260926 13:17:49.388907 16323 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:49.389002 16323 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:17:49.389726 16323 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Sending DeleteTablet for 3 replicas of tablet f25abd4284db49158ab9637ea76ad485
I20260926 13:17:49.390059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 16361
I20260926 13:17:49.390053 16445 tablet_service.cc:1514] Processing DeleteTablet for tablet f25abd4284db49158ab9637ea76ad485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40284
I20260926 13:17:49.390057 16583 tablet_service.cc:1514] Processing DeleteTablet for tablet f25abd4284db49158ab9637ea76ad485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57032
I20260926 13:17:49.390928 16715 tablet_service.cc:1514] Processing DeleteTablet for tablet f25abd4284db49158ab9637ea76ad485 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:17:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36420
I20260926 13:17:49.391053 16811 tablet_replica.cc:330] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: stopping tablet replica
I20260926 13:17:49.391156 16811 raft_consensus.cc:2229] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:17:49.391278 16811 raft_consensus.cc:2258] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:49.391455 16812 tablet_replica.cc:330] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: stopping tablet replica
I20260926 13:17:49.391559 16812 raft_consensus.cc:2229] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:17:49.391652 16812 raft_consensus.cc:2258] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:17:49.391964 16811 ts_tablet_manager.cc:1905] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:49.392056 16812 ts_tablet_manager.cc:1905] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:17:49.393940 16811 ts_tablet_manager.cc:1918] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:49.393977 16812 ts_tablet_manager.cc:1918] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:17:49.394049 16812 log.cc:1196] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/f25abd4284db49158ab9637ea76ad485
I20260926 13:17:49.394104 16811 log.cc:1196] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/f25abd4284db49158ab9637ea76ad485
I20260926 13:17:49.394444 16812 ts_tablet_manager.cc:1939] T f25abd4284db49158ab9637ea76ad485 P d690e2debf9e487e873665490ef0e2d9: Deleting consensus metadata
I20260926 13:17:49.394570 16811 ts_tablet_manager.cc:1939] T f25abd4284db49158ab9637ea76ad485 P eb7436a4b5f549a49b84a6d67c2dc02b: Deleting consensus metadata
I20260926 13:17:49.394860 16310 catalog_manager.cc:4841] TS d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): tablet f25abd4284db49158ab9637ea76ad485 (table test-table [id=34eec282e3844372b1fd845e8447374a]) successfully deleted
I20260926 13:17:49.394986 16309 catalog_manager.cc:4841] TS eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): tablet f25abd4284db49158ab9637ea76ad485 (table test-table [id=34eec282e3844372b1fd845e8447374a]) successfully deleted
W20260926 13:17:49.397063 16310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36485 (error 108)
I20260926 13:17:49.397219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 16513
I20260926 13:17:49.398411 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:17:49.398287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36485 (local address 127.0.0.1:40294)
0926 13:17:49.398339 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:17:49.398378 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:17:49.398526 16310 catalog_manager.cc:4584] TS 53320c4025df4910a073fa58d822917c (127.0.98.1:36485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f25abd4284db49158ab9637ea76ad485: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36485: connect: Connection refused (error 111)
I20260926 13:17:49.402413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 16652
I20260926 13:17:49.407101 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 16290
I20260926 13:18:09.413013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45983
--webserver_interface=127.0.98.62
--webserver_port=35801
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:09.494123 16814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:09.494354 16814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:09.494376 16814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:09.495594 16814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 13:18:09.495640 16814 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 13:18:09.495654 16814 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 13:18:09.495666 16814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:09.495678 16814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:09.495692 16814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:09.495739 16814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:09.495761 16814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:09.497319 16814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:09.497584 16814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:09.497764 16814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:18:09.503091 16814 server_base.cc:1034] running on GCE node
W20260926 13:18:09.503083 16822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.503052 16819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.503060 16820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:09.503499 16814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:09.503818 16814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:09.504974 16814 hybrid_clock.cc:648] HybridClock initialized: now 1790428689504963 us; error 33 us; skew 500 ppm
Sep 26 13:18:09 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:09.515256 16814 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:09.515741 16814 webserver.cc:460] Webserver started at http://127.0.98.62:35801/ using document root <none> and password file <none>
I20260926 13:18:09.515993 16814 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:09.516068 16814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:09.517441 16814 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:09.518141 16829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.518383 16814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:09.518487 16814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "ba7d7675d5bf48cca8c9c786fd9aa395"
format_stamp: "Formatted at 2026-09-26 13:17:27 on dist-test-slave-f864"
server_key: "c49037d54c5930fee495b5d2090d20ff"
server_key_iv: "d40ef71a1fd24cbc7520383aaa65cc1b"
server_key_version: "encryptionkey@0"
I20260926 13:18:09.518868 16814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:09.528213 16814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:09.530041 16814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:09.530264 16814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:09.534003 16814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45983
I20260926 13:18:09.534056 16881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45983 every 8 connection(s)
I20260926 13:18:09.534480 16814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:09.536012 16882 sys_catalog.cc:263] Verifying existing consensus state
I20260926 13:18:09.536590 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap starting.
I20260926 13:18:09.538101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16814
I20260926 13:18:09.538511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36485
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45983
--webserver_port=37927
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:18:09.545660 16882 log.cc:824] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:09.548164 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:18:09.548503 16882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Bootstrap complete.
I20260926 13:18:09.550863 16882 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:18:09.551086 16882 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7d7675d5bf48cca8c9c786fd9aa395, State: Initialized, Role: FOLLOWER
I20260926 13:18:09.551260 16882 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:18:09.551319 16882 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:09.551384 16882 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:09.551431 16882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 2 FOLLOWER]: Advancing to term 3
I20260926 13:18:09.552500 16882 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:18:09.552627 16882 leader_election.cc:302] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba7d7675d5bf48cca8c9c786fd9aa395; no voters:
I20260926 13:18:09.552847 16882 leader_election.cc:288] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 13:18:09.552883 16886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 13:18:09.552996 16886 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [term 3 LEADER]: Becoming Leader. State: Replica: ba7d7675d5bf48cca8c9c786fd9aa395, State: Running, Role: LEADER
I20260926 13:18:09.553123 16882 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:09.553153 16886 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } }
I20260926 13:18:09.553519 16888 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba7d7675d5bf48cca8c9c786fd9aa395. Latest consensus state: current_term: 3 leader_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } } }
I20260926 13:18:09.553550 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba7d7675d5bf48cca8c9c786fd9aa395" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45983 } } }
I20260926 13:18:09.553647 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:09.553602 16888 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:09.553926 16893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:09.554850 16893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:09.555007 16893 catalog_manager.cc:1259] Loaded cluster ID: 5a1fccd1035541a88df83e369a5424e2
I20260926 13:18:09.555263 16893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:09.556461 16893 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:09.556690 16893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Loaded TSK: 0
I20260926 13:18:09.556787 16893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ba7d7675d5bf48cca8c9c786fd9aa395: Loaded TSK: 1
F20260926 13:18:09.557103 16893 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff0ca3769ad google::LogMessage::Fail() at ??:0
@ 0x7ff0ca37988a google::LogMessage::SendToLog() at ??:0
@ 0x7ff0ca3764de google::LogMessage::Flush() at ??:0
@ 0x7ff0ca377a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff0ccae3104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff0ccae0d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff0d22d01cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff0d22d048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff0d22d0594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff0d2296c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff0d22a0ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff0cc2440d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff0cc239954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff0cd09d6db start_thread at ??:0
@ 0x7ff0c8e7461f clone at ??:0
W20260926 13:18:09.638746 16884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:09.638949 16884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:09.638974 16884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:09.640133 16884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:09.640179 16884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:09.640206 16884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:09.641640 16884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37927
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:09.641872 16884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:09.642114 16884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:09.648370 16924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.648345 16926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.648342 16923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:09.648648 16884 server_base.cc:1034] running on GCE node
I20260926 13:18:09.648852 16884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:09.649113 16884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:09.650233 16884 hybrid_clock.cc:648] HybridClock initialized: now 1790428689650216 us; error 34 us; skew 500 ppm
Sep 26 13:18:09 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:09.652858 16884 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:18:09.653318 16884 webserver.cc:460] Webserver started at http://127.0.98.1:37927/ using document root <none> and password file <none>
I20260926 13:18:09.653556 16884 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:09.653633 16884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:09.655028 16884 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:09.655714 16933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.655915 16884 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:09.656023 16884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "53320c4025df4910a073fa58d822917c"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "002fa341ff7d5cd6fc5ad0c11c157ef7"
server_key_iv: "e93160ede48356596db0139d64ede1df"
server_key_version: "encryptionkey@0"
I20260926 13:18:09.656507 16884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:09.679040 16884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:09.681703 16884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:09.681869 16884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:09.682149 16884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:09.682703 16940 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 13:18:09.684283 16884 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 13:18:09.684342 16884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 13:18:09.684378 16884 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 13:18:09.685249 16884 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 13:18:09.685295 16884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 13:18:09.685381 16940 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Bootstrap starting.
I20260926 13:18:09.692366 16884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36485
I20260926 13:18:09.692830 16884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:09.692824 17047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36485 every 8 connection(s)
I20260926 13:18:09.694144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 16884
I20260926 13:18:09.694525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36391
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45983
--webserver_port=35269
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:18:09.696254 16940 log.cc:824] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:09.698611 16940 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 13:18:09.698942 16940 tablet_bootstrap.cc:492] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Bootstrap complete.
I20260926 13:18:09.700075 16940 ts_tablet_manager.cc:1397] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20260926 13:18:09.701879 16940 raft_consensus.cc:348] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:09.702287 16940 raft_consensus.cc:729] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53320c4025df4910a073fa58d822917c, State: Initialized, Role: FOLLOWER
I20260926 13:18:09.702473 16940 consensus_queue.cc:260] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:09.702798 16940 ts_tablet_manager.cc:1428] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
W20260926 13:18:09.758213 17050 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:09.693678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:46417)
0926 13:18:09.693901 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.693968 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:09.696226 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:09.757978 (+ 61752us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":101,"threads_started":1}
W20260926 13:18:09.758491 17048 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45983 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45983: Client connection negotiation failed: client connection to 127.0.98.62:45983: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 13:18:09.758837 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.758714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:37365)
0926 13:18:09.758771 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.758814 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 13:18:09.759225 17050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.759135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:38011)
0926 13:18:09.759179 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.759205 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:18:09.759338 17048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:18:09.779667 17051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:09.779867 17051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:09.779889 17051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:09.781020 17051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:09.781061 17051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:09.781087 17051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:09.782588 17051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35269
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:09.782824 17051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:09.783052 17051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:09.788134 17061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.788122 17058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.788123 17059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:09.788303 17051 server_base.cc:1034] running on GCE node
I20260926 13:18:09.788573 17051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:09.788805 17051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:09.789973 17051 hybrid_clock.cc:648] HybridClock initialized: now 1790428689789951 us; error 34 us; skew 500 ppm
Sep 26 13:18:09 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:09.792476 17051 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:18:09.792944 17051 webserver.cc:460] Webserver started at http://127.0.98.2:35269/ using document root <none> and password file <none>
I20260926 13:18:09.793208 17051 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:09.793282 17051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:09.794762 17051 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:18:09.795408 17068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.795606 17051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:09.795702 17051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "eb7436a4b5f549a49b84a6d67c2dc02b"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "9c7dd297bba3755f37dc93acb3f1aa3e"
server_key_iv: "e33c36a177171ee132b503f7da7c0f75"
server_key_version: "encryptionkey@0"
I20260926 13:18:09.796017 17051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:09.811724 17051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:09.813999 17051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:09.814155 17051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:09.814422 17051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:09.814723 17051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:09.814774 17051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.814813 17051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:09.814841 17051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.820705 17051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36391
I20260926 13:18:09.820742 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36391 every 8 connection(s)
I20260926 13:18:09.821094 17051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:18:09.822065 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17051
I20260926 13:18:09.822093 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.821745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:34465)
0926 13:18:09.821896 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.821961 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260926 13:18:09.822376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46405
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45983
--webserver_port=35711
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:09.822616 17182 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45983 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45983: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
I20260926 13:18:09.822938 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.822852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:39773)
0926 13:18:09.822896 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.822921 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 13:18:09.823238 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.823151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:38349)
0926 13:18:09.823194 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.823222 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:18:09.823333 17182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 13:18:09.903304 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:09.903551 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:09.903592 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:09.904783 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:09.904858 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:09.904903 17185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:09.906528 17185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46405
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35711
--tserver_master_addrs=127.0.98.62:45983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:09.906810 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:09.907075 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:09.912245 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:09.912294 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:09.912439 17185 server_base.cc:1034] running on GCE node
W20260926 13:18:09.912245 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:09.912738 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:09.913017 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:09.914232 17185 hybrid_clock.cc:648] HybridClock initialized: now 1790428689914200 us; error 40 us; skew 500 ppm
Sep 26 13:18:09 dist-test-slave-f864 krb5kdc[15744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:09.916747 17185 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:18:09.917209 17185 webserver.cc:460] Webserver started at http://127.0.98.3:35711/ using document root <none> and password file <none>
I20260926 13:18:09.917441 17185 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:09.917512 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:09.918951 17185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:09.919553 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.919723 17185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:09.919828 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "d690e2debf9e487e873665490ef0e2d9"
format_stamp: "Formatted at 2026-09-26 13:17:28 on dist-test-slave-f864"
server_key: "c72704e826906db7861299e2d7fd6ff4"
server_key_iv: "b03b3c80857877c321bc0015627e372d"
server_key_version: "encryptionkey@0"
I20260926 13:18:09.920192 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:09.949569 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:09.951561 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:09.951737 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:09.951978 17185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:09.952301 17185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:09.952358 17185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.952399 17185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:09.952427 17185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:09.958278 17185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46405
I20260926 13:18:09.958352 17313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46405 every 8 connection(s)
I20260926 13:18:09.958717 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:18:09.959800 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.959435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:59587)
0926 13:18:09.959577 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.959645 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
W20260926 13:18:09.960213 17314 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45983 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45983: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
I20260926 13:18:09.960495 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.960410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:38955)
0926 13:18:09.960452 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.960479 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 13:18:09.960726 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:09.960675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:52833)
0926 13:18:09.960696 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:09.960714 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 13:18:09.960827 17314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 13:18:09.967520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17185
I20260926 13:18:10.759878 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:10.759657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:42871)
0926 13:18:10.759781 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:10.759816 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260926 13:18:10.823917 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:10.823687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:40393)
0926 13:18:10.823832 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:10.823867 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260926 13:18:10.961318 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:10.961116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:55895)
0926 13:18:10.961241 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:10.961274 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
I20260926 13:18:11.266865 17321 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:11.266983 17321 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:11.267287 17321 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
Sep 26 13:18:11 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 13:18:11 dist-test-slave-f864 krb5kdc[15744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 13:18:11.278057 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.267578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37915 (local address 127.0.98.3:46405)
0926 13:18:11.267763 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:11.267775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:11.267809 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:18:11.268387 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:11.268390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:11.268449 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:11.268605 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:11.269349 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.270272 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.271257 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.271653 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.271786 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:11.274274 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:11.274278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:11.274278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:11.274288 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:11.276342 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:11.276782 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:11.276785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:11.276786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:11.276882 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:11.277148 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:11.277149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:11.277149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:11.277391 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:11.277479 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:11.277795 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 13:18:11.277922 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":54,"threads_started":1}
I20260926 13:18:11.278062 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.267586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41827 (local address 127.0.98.2:36391)
0926 13:18:11.267763 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:11.267775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:11.267809 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:18:11.268387 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:11.268390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:11.268449 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:11.268605 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:11.269348 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.270289 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.271245 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.271653 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.271793 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:11.274729 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:11.274732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:11.274732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:11.274742 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:11.276596 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:11.276940 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:11.276943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:11.276944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:11.277032 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:11.277236 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:11.277237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:11.277238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:11.277461 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:11.277546 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:11.277794 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 13:18:11.277940 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":67,"threads_started":1}
I20260926 13:18:11.278056 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.267432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46405 (local address 127.0.98.1:37915)
0926 13:18:11.267552 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:11.267564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:11.267623 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:11.268754 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:11.268755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:11.269017 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:11.269220 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:11.269226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.270369 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.270370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:11.271150 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:11.271152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.271760 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.271761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:11.271884 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:11.272524 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:11.272529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:11.274131 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:11.276437 (+ 2306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:11.276443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:11.276445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:11.276667 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:11.276956 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:11.276957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:11.276957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:11.277069 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:11.277473 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:11.277479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:11.277627 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 13:18:11.277731 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 13:18:11.277920 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":10,"thread_start_us":63,"threads_started":1}
I20260926 13:18:11.278373 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.267463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36391 (local address 127.0.98.1:41827)
0926 13:18:11.267587 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:11.267598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:11.267655 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:11.268679 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:11.268681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:11.269016 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:11.269220 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:11.269226 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.270392 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.270393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:11.271150 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:11.271152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:11.271759 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:11.271760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:11.271888 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:11.272460 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:11.272468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:11.274625 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:11.276685 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:11.276686 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:11.276687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:11.276857 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:11.277104 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:11.277105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:11.277105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:11.277169 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:11.277542 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:11.277543 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:11.277627 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 13:18:11.277730 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:18:11.278003 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":195,"thread_start_us":55,"threads_started":1}
W20260926 13:18:11.279017 16937 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
W20260926 13:18:11.279114 16936 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
I20260926 13:18:11.279182 16936 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53320c4025df4910a073fa58d822917c; no voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b
I20260926 13:18:11.279289 17321 raft_consensus.cc:2726] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:18:11.760473 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.760336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:39117)
0926 13:18:11.760393 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:11.760433 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:18:11.824577 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.824390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:42111)
0926 13:18:11.824505 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:11.824542 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 13:18:11.961915 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:11.961717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:38327)
0926 13:18:11.961840 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:11.961876 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":57,"threads_started":1}
I20260926 13:18:12.761212 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:12.760970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:51357)
0926 13:18:12.761122 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:12.761164 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260926 13:18:12.825225 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:12.825029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:33591)
0926 13:18:12.825148 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:12.825185 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260926 13:18:12.962637 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:12.962423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:43887)
0926 13:18:12.962553 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:12.962594 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260926 13:18:12.968905 17331 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:12.968999 17331 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:12.969156 17331 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
W20260926 13:18:12.969568 16937 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
W20260926 13:18:12.969640 16936 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
I20260926 13:18:12.969710 16936 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53320c4025df4910a073fa58d822917c; no voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b
I20260926 13:18:12.969803 17331 raft_consensus.cc:2726] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:18:13.761981 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:13.761775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:44435)
0926 13:18:13.761904 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:13.761933 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260926 13:18:13.825811 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:13.825615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:44695)
0926 13:18:13.825733 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:13.825772 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260926 13:18:13.963301 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:13.963062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:60489)
0926 13:18:13.963222 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:13.963261 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":60,"threads_started":1}
I20260926 13:18:14.762789 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:14.762562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:48813)
0926 13:18:14.762697 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:14.762743 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260926 13:18:14.826572 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:14.826356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:55687)
0926 13:18:14.826479 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:14.826534 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260926 13:18:14.963999 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:14.963786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:33821)
0926 13:18:14.963910 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:14.963955 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":56,"threads_started":1}
I20260926 13:18:15.418012 17338 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:15.418100 17338 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:15.418347 17338 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
W20260926 13:18:15.418740 16937 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
W20260926 13:18:15.418845 16936 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
I20260926 13:18:15.418915 16936 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53320c4025df4910a073fa58d822917c; no voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b
I20260926 13:18:15.419020 17338 raft_consensus.cc:2726] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:18:15.763455 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:15.763242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:43619)
0926 13:18:15.763378 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:15.763408 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260926 13:18:15.827188 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:15.826971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:41871)
0926 13:18:15.827114 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:15.827148 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260926 13:18:15.964644 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:15.964428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:39699)
0926 13:18:15.964562 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:15.964600 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260926 13:18:16.764253 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:16.764044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:55035)
0926 13:18:16.764166 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:16.764205 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 13:18:16.827821 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:16.827617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:33921)
0926 13:18:16.827746 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:16.827780 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260926 13:18:16.965356 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:16.965122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:57503)
0926 13:18:16.965250 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:16.965299 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260926 13:18:17.513484 17345 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:17.513586 17345 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:17.513732 17345 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
W20260926 13:18:17.514320 16936 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
W20260926 13:18:17.514407 16937 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
I20260926 13:18:17.514508 16937 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53320c4025df4910a073fa58d822917c; no voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b
I20260926 13:18:17.514613 17345 raft_consensus.cc:2726] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:18:17.765026 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:17.764802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:58129)
0926 13:18:17.764941 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:17.764980 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
I20260926 13:18:17.828495 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:17.828260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:40025)
0926 13:18:17.828395 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:17.828438 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260926 13:18:17.966109 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:17.965842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:59401)
0926 13:18:17.966008 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:17.966050 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":125,"threads_started":1}
I20260926 13:18:18.765694 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:18.765470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:43569)
0926 13:18:18.765613 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:18.765646 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260926 13:18:18.829159 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:18.828939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:53867)
0926 13:18:18.829080 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:18.829113 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 13:18:18.966796 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:18.966575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:48601)
0926 13:18:18.966715 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:18.966752 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260926 13:18:19.300222 17352 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:19.300344 17352 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:19.300534 17352 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
W20260926 13:18:19.300951 16936 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
W20260926 13:18:19.301079 16937 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
I20260926 13:18:19.301143 16937 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53320c4025df4910a073fa58d822917c; no voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b
I20260926 13:18:19.301267 17352 raft_consensus.cc:2726] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:18:19.766615 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:19.766375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:56677)
0926 13:18:19.766527 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:19.766568 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260926 13:18:19.829928 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:19.829731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:33977)
0926 13:18:19.829851 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:19.829886 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260926 13:18:19.967522 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:19.967291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:55663)
0926 13:18:19.967426 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:19.967464 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260926 13:18:20.767324 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:20.767092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:46233)
0926 13:18:20.767241 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:20.767276 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260926 13:18:20.830634 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:20.830378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:59257)
0926 13:18:20.830527 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:20.830564 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260926 13:18:20.968256 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:20.968038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:60095)
0926 13:18:20.968178 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:20.968211 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260926 13:18:21.151222 17359 raft_consensus.cc:482] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:21.151314 17359 raft_consensus.cc:504] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d690e2debf9e487e873665490ef0e2d9" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46405 } } peers { permanent_uuid: "eb7436a4b5f549a49b84a6d67c2dc02b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36391 } } peers { permanent_uuid: "53320c4025df4910a073fa58d822917c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36485 } }
I20260926 13:18:21.151633 17359 leader_election.cc:288] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405), eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391)
W20260926 13:18:21.152045 16937 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d690e2debf9e487e873665490ef0e2d9 (127.0.98.3:46405): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
W20260926 13:18:21.152132 16936 leader_election.cc:341] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb7436a4b5f549a49b84a6d67c2dc02b (127.0.98.2:36391): Not found: Tablet not found: f25abd4284db49158ab9637ea76ad485
I20260926 13:18:21.152207 16936 leader_election.cc:302] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53320c4025df4910a073fa58d822917c; no voters: d690e2debf9e487e873665490ef0e2d9, eb7436a4b5f549a49b84a6d67c2dc02b
I20260926 13:18:21.152266 17359 raft_consensus.cc:2726] T f25abd4284db49158ab9637ea76ad485 P 53320c4025df4910a073fa58d822917c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 13:18:21.768162 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:21.767937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:33867)
0926 13:18:21.768063 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:21.768104 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260926 13:18:21.831342 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:21.831132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:47291)
0926 13:18:21.831258 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:21.831297 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260926 13:18:21.969069 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:21.968855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:59909)
0926 13:18:21.968980 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:21.969023 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260926 13:18:22.768915 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:22.768599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:43455)
0926 13:18:22.768832 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:22.768870 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":74,"threads_started":1}
I20260926 13:18:22.831977 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:22.831770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:37931)
0926 13:18:22.831900 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:22.831935 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260926 13:18:22.969765 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:22.969530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:55187)
0926 13:18:22.969678 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:22.969719 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260926 13:18:23.769614 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:23.769392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:42145)
0926 13:18:23.769517 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:23.769559 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260926 13:18:23.832695 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:23.832461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:42647)
0926 13:18:23.832609 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:23.832649 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260926 13:18:23.970453 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:23.970247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:52949)
0926 13:18:23.970367 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:23.970407 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260926 13:18:24.770352 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:24.770112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.1:39953)
0926 13:18:24.770281 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:24.770309 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":111,"threads_started":1}
I20260926 13:18:24.833410 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:24.833213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.2:51565)
0926 13:18:24.833333 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:24.833366 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260926 13:18:24.971112 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:24.970931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45983 (local address 127.0.98.3:49261)
0926 13:18:24.971039 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:24.971072 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260926 13:18:24.976418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 16884
I20260926 13:18:24.981081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17051
I20260926 13:18:24.985405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17185
2026-09-26T13:18:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58420 ms)
[----------] 30 tests from SecurityITest (260697 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:25 dist-test-slave-f864 krb5kdc[17374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:25 dist-test-slave-f864 krb5kdc[17374](info): set up 2 sockets
Sep 26 13:18:25 dist-test-slave-f864 krb5kdc[17374](info): commencing operation
krb5kdc: starting...
W20260926 13:18:26.009765 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:26.121215 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45099
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:26.198951 17387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:26.199185 17387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:26.199223 17387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:26.200343 17387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:26.200412 17387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:26.200434 17387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:26.200456 17387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:26.200482 17387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:26.201869 17387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:26.202131 17387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:26.202350 17387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:26.207430 17395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.207504 17387 server_base.cc:1034] running on GCE node
W20260926 13:18:26.207407 17393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.207408 17392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.207835 17387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:26.208098 17387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:26.209260 17387 hybrid_clock.cc:648] HybridClock initialized: now 1790428706209246 us; error 27 us; skew 500 ppm
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:26.211652 17387 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:26.212072 17387 webserver.cc:460] Webserver started at http://127.0.98.62:38999/ using document root <none> and password file <none>
I20260926 13:18:26.212289 17387 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:26.212353 17387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:26.212491 17387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:26.213474 17387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "5af8145f3512420891a7c80b7a6b46ed"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "05f7c0b755b93ec568911d556ceb94c2"
server_key_iv: "86d54735c453aa5bda6304e951bd1eaa"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.213822 17387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "5af8145f3512420891a7c80b7a6b46ed"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "05f7c0b755b93ec568911d556ceb94c2"
server_key_iv: "86d54735c453aa5bda6304e951bd1eaa"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.215114 17387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:18:26.215799 17402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.215952 17387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.216046 17387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "5af8145f3512420891a7c80b7a6b46ed"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "05f7c0b755b93ec568911d556ceb94c2"
server_key_iv: "86d54735c453aa5bda6304e951bd1eaa"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.216114 17387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:26.232874 17387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:26.234789 17387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:26.235041 17387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:26.239161 17387 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45099
I20260926 13:18:26.239189 17454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45099 every 8 connection(s)
I20260926 13:18:26.239531 17387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:26.240332 17455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:26.242573 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: Bootstrap starting.
I20260926 13:18:26.243119 17455 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:26.243391 17455 log.cc:824] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:26.244134 17455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: No bootstrap required, opened a new log
I20260926 13:18:26.245546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17387
I20260926 13:18:26.245626 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:26.245827 392 external_mini_cluster.cc:1428] Setting key 2fddea9d7f9314ef42bb377f46c1bee8
I20260926 13:18:26.245831 17455 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8145f3512420891a7c80b7a6b46ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260926 13:18:26.246011 17455 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:26.246058 17455 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af8145f3512420891a7c80b7a6b46ed, State: Initialized, Role: FOLLOWER
I20260926 13:18:26.246224 17455 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8145f3512420891a7c80b7a6b46ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260926 13:18:26.246417 17455 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:26.246481 17455 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:26.246536 17455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:26.247392 17455 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8145f3512420891a7c80b7a6b46ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260926 13:18:26.247509 17455 leader_election.cc:302] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af8145f3512420891a7c80b7a6b46ed; no voters:
I20260926 13:18:26.247666 17455 leader_election.cc:288] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:26.247697 17460 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:26.247799 17460 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [term 1 LEADER]: Becoming Leader. State: Replica: 5af8145f3512420891a7c80b7a6b46ed, State: Running, Role: LEADER
I20260926 13:18:26.247922 17455 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:26.247931 17460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8145f3512420891a7c80b7a6b46ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } }
I20260926 13:18:26.248346 17462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af8145f3512420891a7c80b7a6b46ed. Latest consensus state: current_term: 1 leader_uuid: "5af8145f3512420891a7c80b7a6b46ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8145f3512420891a7c80b7a6b46ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } } }
I20260926 13:18:26.248476 17462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:26.248642 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af8145f3512420891a7c80b7a6b46ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8145f3512420891a7c80b7a6b46ed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45099 } } }
I20260926 13:18:26.248735 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed [sys.catalog]: This master's current role is: LEADER
W20260926 13:18:26.249804 17476 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:18:26.249857 17476 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:18:26.249907 17467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:26.250065 17467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:26.251458 17467 catalog_manager.cc:1347] Generated new cluster ID: 601315f83aa34d69b5542d1ccd3744a0
I20260926 13:18:26.251513 17467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:26.256909 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.246272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.0.98.62:45099)
0926 13:18:26.246455 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.246465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.246502 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:18:26.246617 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.246637 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.246685 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.246811 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:26.247428 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.248812 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.249773 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.250371 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.250482 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.253656 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:26.253660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:26.253660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:26.253671 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:26.255376 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.255741 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.255744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.255745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.255841 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.256042 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.256043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.256044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.256278 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:26.256435 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.256528 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.256767 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260926 13:18:26.276544 17467 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:26.277287 17467 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:26.286067 17467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: Generated new TSK 0
I20260926 13:18:26.286270 17467 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:18:26.320720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:26.404700 17481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:26.404891 17481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:26.404914 17481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:26.406204 17481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:26.406246 17481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:26.406288 17481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:26.407778 17481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:26.408022 17481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:26.408249 17481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:26.413371 17486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.413324 17489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.413329 17487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.413669 17481 server_base.cc:1034] running on GCE node
I20260926 13:18:26.413887 17481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:26.414135 17481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:26.415323 17481 hybrid_clock.cc:648] HybridClock initialized: now 1790428706415311 us; error 29 us; skew 500 ppm
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:26.417840 17481 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:18:26.418349 17481 webserver.cc:460] Webserver started at http://127.0.98.1:35271/ using document root <none> and password file <none>
I20260926 13:18:26.418632 17481 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:26.418699 17481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:26.418845 17481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:26.420120 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "722d3689e8ad4f559a2332f40d184c04"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "29a00320cd228d10870f514fe756f27a"
server_key_iv: "4f792ae3f24138780dc13286be3177c7"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.420514 17481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "722d3689e8ad4f559a2332f40d184c04"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "29a00320cd228d10870f514fe756f27a"
server_key_iv: "4f792ae3f24138780dc13286be3177c7"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.422005 17481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:18:26.422822 17496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.423009 17481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:18:26.423079 17481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "722d3689e8ad4f559a2332f40d184c04"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "29a00320cd228d10870f514fe756f27a"
server_key_iv: "4f792ae3f24138780dc13286be3177c7"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.423131 17481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:26.445047 17481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:26.447328 17481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:26.447458 17481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:26.447639 17481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:26.447986 17481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:26.448024 17481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.448048 17481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:26.448074 17481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.454217 17481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42371
I20260926 13:18:26.454264 17609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42371 every 8 connection(s)
I20260926 13:18:26.454600 17481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:26.455353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17481
I20260926 13:18:26.455487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:18:26.455699 392 external_mini_cluster.cc:1428] Setting key 038a290ae708a73aad257b65cd7cd850
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:18:26.465391 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.455358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40547 (local address 127.0.98.62:45099)
0926 13:18:26.455423 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.455425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.455673 (+ 248us) server_negotiation.cc:407] Connection header received
0926 13:18:26.456751 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.456752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.456769 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.456840 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:26.457488 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.458069 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.459697 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.459950 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.460004 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.462515 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:26.462516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:26.462517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:26.462525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:26.464206 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.464643 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.464644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.464645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.464705 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.464909 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.464910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.464910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.465014 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:26.465065 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.465269 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.465328 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:18:26.465626 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.455316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.1:40547)
0926 13:18:26.455544 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.455589 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.456650 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.456905 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.456908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.457196 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:26.457390 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.457395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.458157 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.458180 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.459598 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.459602 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.460048 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.460050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.460206 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.460798 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:26.460806 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:26.462425 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:26.464293 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:26.464299 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:26.464300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:26.464573 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:26.464773 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:26.464773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:26.464774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:26.464867 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:26.465065 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:26.465069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:26.465179 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.465258 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.465368 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260926 13:18:26.466195 17610 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45099
I20260926 13:18:26.466279 17610 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:26.466418 17610 heartbeater.cc:511] Master 127.0.98.62:45099 requested a full tablet report, sending...
I20260926 13:18:26.466965 17419 ts_manager.cc:194] Registered new tserver with Master: 722d3689e8ad4f559a2332f40d184c04 (127.0.98.1:42371)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 13:18:26.467783 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:18:26.481186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:26.560611 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:26.560844 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:26.560879 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:26.562022 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:26.562083 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:26.562127 17615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:26.563628 17615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:26.563881 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:26.564095 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:26.568889 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.568926 17615 server_base.cc:1034] running on GCE node
W20260926 13:18:26.568882 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.568883 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.569250 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:26.569499 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:26.570659 17615 hybrid_clock.cc:648] HybridClock initialized: now 1790428706570644 us; error 31 us; skew 500 ppm
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:26.572979 17615 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:18:26.573431 17615 webserver.cc:460] Webserver started at http://127.0.98.2:40185/ using document root <none> and password file <none>
I20260926 13:18:26.573655 17615 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:26.573720 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:26.573853 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:26.574920 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "ad6e1a4011cd4d3db056298f9eba39c8"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "7d813493b11deebedf179596da9b11d5"
server_key_iv: "57ba44dfb3656ed7265ff29b6cd1c3b5"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.575274 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ad6e1a4011cd4d3db056298f9eba39c8"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "7d813493b11deebedf179596da9b11d5"
server_key_iv: "57ba44dfb3656ed7265ff29b6cd1c3b5"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.576516 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:26.577194 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.577347 17615 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:26.577436 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "ad6e1a4011cd4d3db056298f9eba39c8"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "7d813493b11deebedf179596da9b11d5"
server_key_iv: "57ba44dfb3656ed7265ff29b6cd1c3b5"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.577510 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:26.594086 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:26.595836 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:26.595986 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:26.596199 17615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:26.596484 17615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:26.596539 17615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.596580 17615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:26.596609 17615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.602475 17615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34525
I20260926 13:18:26.602519 17743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34525 every 8 connection(s)
I20260926 13:18:26.602828 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:18:26.605403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17615
I20260926 13:18:26.605508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:18:26.605708 392 external_mini_cluster.cc:1428] Setting key 57ab1eb99b37c494f53dbfbcf0b13bff
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:26.613327 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.603478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53749 (local address 127.0.98.62:45099)
0926 13:18:26.603526 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.603528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.603700 (+ 172us) server_negotiation.cc:407] Connection header received
0926 13:18:26.604655 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.604657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.604670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.604743 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:26.605355 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.606076 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.607655 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.607981 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.608057 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.610241 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:26.610242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:26.610243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:26.610250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:26.611936 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.612309 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.612311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.612311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.612377 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.612645 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.612646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.612647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.612792 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:26.612866 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.613122 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.613201 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:18:26.613338 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.603442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.2:53749)
0926 13:18:26.603577 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.603627 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.604530 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.604808 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.604812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.605065 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:26.605262 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.605266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.606204 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.606205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.607568 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.607571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.608068 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.608070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.608177 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.608701 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:26.608710 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:26.610115 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:26.612013 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:26.612020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:26.612027 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:26.612234 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:26.612445 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:26.612446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:26.612446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:26.612572 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:26.612855 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:26.612860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:26.612968 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.613068 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.613183 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 13:18:26.614152 17744 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45099
I20260926 13:18:26.614277 17744 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:26.614472 17744 heartbeater.cc:511] Master 127.0.98.62:45099 requested a full tablet report, sending...
I20260926 13:18:26.614810 17419 ts_manager.cc:194] Registered new tserver with Master: ad6e1a4011cd4d3db056298f9eba39c8 (127.0.98.2:34525)
I20260926 13:18:26.615190 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:18:26.632428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:26.711207 17749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:26.711404 17749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:26.711427 17749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:26.712610 17749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:26.712649 17749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:26.712675 17749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:26.714043 17749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:26.714277 17749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:26.714537 17749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:26.719535 17754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.719554 17755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.719535 17757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.720001 17749 server_base.cc:1034] running on GCE node
I20260926 13:18:26.720242 17749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:26.720472 17749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:26.721640 17749 hybrid_clock.cc:648] HybridClock initialized: now 1790428706721632 us; error 36 us; skew 500 ppm
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:26.724155 17749 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:18:26.724612 17749 webserver.cc:460] Webserver started at http://127.0.98.3:41259/ using document root <none> and password file <none>
I20260926 13:18:26.724834 17749 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:26.724897 17749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:26.725034 17749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:26.726253 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "ead88e6014da4f7ca058882a0aaa10af"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "f40222c869d17386282aff250173b4dc"
server_key_iv: "18ed2255e6ccda82a268d015f96dab35"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.726671 17749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ead88e6014da4f7ca058882a0aaa10af"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "f40222c869d17386282aff250173b4dc"
server_key_iv: "18ed2255e6ccda82a268d015f96dab35"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.728011 17749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:18:26.728694 17764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.728852 17749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:26.728947 17749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "ead88e6014da4f7ca058882a0aaa10af"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "f40222c869d17386282aff250173b4dc"
server_key_iv: "18ed2255e6ccda82a268d015f96dab35"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.729014 17749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:26.739431 17749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:26.741182 17749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:26.741334 17749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:26.741547 17749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:26.741854 17749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:26.741904 17749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.741943 17749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:26.741971 17749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.748410 17877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40475 every 8 connection(s)
I20260926 13:18:26.748463 17749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40475
I20260926 13:18:26.748829 17749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:18:26 dist-test-slave-f864 krb5kdc[17374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428706, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:26.757001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17749
I20260926 13:18:26.757117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:18:26.757315 392 external_mini_cluster.cc:1428] Setting key de2808e243fb59ac0200d50f2b599ef6
I20260926 13:18:26.758399 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.749576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44107 (local address 127.0.98.62:45099)
0926 13:18:26.749634 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.749636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.749849 (+ 213us) server_negotiation.cc:407] Connection header received
0926 13:18:26.750882 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.750883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.750896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.750947 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:26.751558 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.752054 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.753058 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.753265 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.753350 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.755468 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:26.755469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:26.755469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:26.755478 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:26.756688 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.757445 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.757447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.757447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.757506 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.757748 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.757749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.757749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.757868 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:26.757939 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.758218 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.758299 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:18:26.758620 17880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.749551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45099 (local address 127.0.98.3:44107)
0926 13:18:26.749734 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.749769 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.750771 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.751012 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.751016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.751270 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:26.751465 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.751469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.752160 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.752161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.752961 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.752963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.753360 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.753361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.753480 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.753954 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:26.753962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:26.755378 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:26.757066 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:26.757073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:26.757075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:26.757337 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:26.757569 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:26.757569 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:26.757570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:26.757683 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:26.757931 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:26.757937 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:26.758074 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.758226 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.758383 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260926 13:18:26.759057 17878 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45099
I20260926 13:18:26.759140 17878 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:26.759281 17878 heartbeater.cc:511] Master 127.0.98.62:45099 requested a full tablet report, sending...
I20260926 13:18:26.759573 17419 ts_manager.cc:194] Registered new tserver with Master: ead88e6014da4f7ca058882a0aaa10af (127.0.98.3:40475)
I20260926 13:18:26.759954 17419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44107
I20260926 13:18:26.769027 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:18:26.775810 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.770370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.0.98.62:45099)
0926 13:18:26.770429 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.770431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.770519 (+ 88us) server_negotiation.cc:407] Connection header received
0926 13:18:26.770580 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.770580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.770588 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.770640 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:26.771147 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.771607 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.772287 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.772451 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.772496 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.773687 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:26.773688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:26.773688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:26.773694 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:26.774908 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.775201 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.775201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.775202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.775248 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:26.775398 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:26.775399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:26.775399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:26.775524 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:26.775594 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.775673 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.775727 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:18:26.777374 17419 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:18:26.779101 17419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:18:26.786201 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.781817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40475 (local address 127.0.0.1:37502)
0926 13:18:26.782636 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.782647 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.782702 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.782906 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.782907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.782909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:26.782963 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.782966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.783951 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.783952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.784956 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.784959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.785843 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.785844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.785929 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.785946 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.786025 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.786086 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":62,"thread_start_us":54,"threads_started":1}
I20260926 13:18:26.786199 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.782028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.3:40475)
0926 13:18:26.782370 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.782374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.782723 (+ 349us) server_negotiation.cc:407] Connection header received
0926 13:18:26.782776 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.782778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.782795 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.782896 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:26.783045 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.783825 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.785068 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.785730 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.785838 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.785905 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.786049 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.786088 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":89,"threads_started":1}
I20260926 13:18:26.786664 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.781717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.0.98.2:34525)
0926 13:18:26.782068 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.782073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.782100 (+ 27us) server_negotiation.cc:407] Connection header received
0926 13:18:26.782154 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.782157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.782205 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.782296 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:26.782617 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.783715 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.785324 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.786050 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.786121 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.786197 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.786548 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.786592 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":85,"threads_started":1}
I20260926 13:18:26.786479 17890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.781591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34525 (local address 127.0.0.1:60414)
0926 13:18:26.781924 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.781955 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.782034 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.782424 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.782427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.782433 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:26.782521 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.782522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.783832 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.783834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.785225 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.785228 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.786231 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.786232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.786293 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.786297 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.786346 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.786386 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":133,"thread_start_us":73,"threads_started":1}
I20260926 13:18:26.787063 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.781586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42371 (local address 127.0.0.1:53284)
0926 13:18:26.781925 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.781958 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.782037 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.782424 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.782427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.782430 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:26.782520 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.782522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.783844 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.783845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.785821 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.785824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.786827 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.786828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.786902 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.786907 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.786955 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.786992 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":910,"thread_start_us":71,"threads_started":1}
I20260926 13:18:26.787336 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.781687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.0.98.1:42371)
0926 13:18:26.781834 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.781844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.782138 (+ 294us) server_negotiation.cc:407] Connection header received
0926 13:18:26.782218 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.782221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.782240 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.782320 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:26.782624 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.783739 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.785918 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.786668 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.786739 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.786794 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.787220 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.787267 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260926 13:18:26.787629 17678 tablet_service.cc:1467] Processing CreateTablet for tablet 293d07e7d8704c8ebcbf926c7fa5cd86 (DEFAULT_TABLE table=test-table [id=2cb5521e519f43a989be4939965cae4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:26.787629 17812 tablet_service.cc:1467] Processing CreateTablet for tablet 293d07e7d8704c8ebcbf926c7fa5cd86 (DEFAULT_TABLE table=test-table [id=2cb5521e519f43a989be4939965cae4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:26.787905 17678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293d07e7d8704c8ebcbf926c7fa5cd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:26.787972 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293d07e7d8704c8ebcbf926c7fa5cd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:26.790482 17892 tablet_bootstrap.cc:492] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Bootstrap starting.
I20260926 13:18:26.790834 17544 tablet_service.cc:1467] Processing CreateTablet for tablet 293d07e7d8704c8ebcbf926c7fa5cd86 (DEFAULT_TABLE table=test-table [id=2cb5521e519f43a989be4939965cae4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:26.790943 17893 tablet_bootstrap.cc:492] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Bootstrap starting.
I20260926 13:18:26.791162 17544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 293d07e7d8704c8ebcbf926c7fa5cd86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:26.791177 17892 tablet_bootstrap.cc:654] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:26.791610 17893 tablet_bootstrap.cc:654] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:26.791757 17892 log.cc:824] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:26.792167 17893 log.cc:824] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:26.792590 17892 tablet_bootstrap.cc:492] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: No bootstrap required, opened a new log
I20260926 13:18:26.792656 17892 ts_tablet_manager.cc:1397] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:26.792871 17893 tablet_bootstrap.cc:492] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: No bootstrap required, opened a new log
I20260926 13:18:26.792922 17893 ts_tablet_manager.cc:1397] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:26.793889 17896 tablet_bootstrap.cc:492] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: Bootstrap starting.
I20260926 13:18:26.794620 17896 tablet_bootstrap.cc:654] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:26.794677 17892 raft_consensus.cc:348] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.794862 17892 raft_consensus.cc:374] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:26.794790 17893 raft_consensus.cc:348] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.794914 17892 raft_consensus.cc:729] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6e1a4011cd4d3db056298f9eba39c8, State: Initialized, Role: FOLLOWER
I20260926 13:18:26.794930 17893 raft_consensus.cc:374] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:26.794962 17893 raft_consensus.cc:729] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead88e6014da4f7ca058882a0aaa10af, State: Initialized, Role: FOLLOWER
I20260926 13:18:26.795051 17892 consensus_queue.cc:260] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.795068 17893 consensus_queue.cc:260] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.795243 17896 log.cc:824] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:26.795318 17893 ts_tablet_manager.cc:1428] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 13:18:26.795318 17892 ts_tablet_manager.cc:1428] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:26.795437 17878 heartbeater.cc:503] Master 127.0.98.62:45099 was elected leader, sending a full tablet report...
I20260926 13:18:26.795473 17744 heartbeater.cc:503] Master 127.0.98.62:45099 was elected leader, sending a full tablet report...
I20260926 13:18:26.796440 17896 tablet_bootstrap.cc:492] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: No bootstrap required, opened a new log
I20260926 13:18:26.796510 17896 ts_tablet_manager.cc:1397] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 13:18:26.797854 17896 raft_consensus.cc:348] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.797981 17896 raft_consensus.cc:374] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:26.798003 17896 raft_consensus.cc:729] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722d3689e8ad4f559a2332f40d184c04, State: Initialized, Role: FOLLOWER
I20260926 13:18:26.798076 17896 consensus_queue.cc:260] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.798364 17896 ts_tablet_manager.cc:1428] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:18:26.798390 17610 heartbeater.cc:503] Master 127.0.98.62:45099 was elected leader, sending a full tablet report...
I20260926 13:18:26.828315 17898 raft_consensus.cc:482] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:26.828428 17898 raft_consensus.cc:504] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.828857 17898 leader_election.cc:288] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 722d3689e8ad4f559a2332f40d184c04 (127.0.98.1:42371), ead88e6014da4f7ca058882a0aaa10af (127.0.98.3:40475)
I20260926 13:18:26.832710 17891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.829096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51109 (local address 127.0.98.3:40475)
0926 13:18:26.829184 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.829187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.829198 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:18:26.829229 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.829230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.829244 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.829315 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:26.829506 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.830292 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.831733 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.832338 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.832405 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.832415 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.832596 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.832642 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 13:18:26.832710 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.829210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36311 (local address 127.0.98.1:42371)
0926 13:18:26.829262 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.829264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.829421 (+ 157us) server_negotiation.cc:407] Connection header received
0926 13:18:26.829450 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.829451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.829462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.829532 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:26.829647 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.830386 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.831733 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.832330 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.832405 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.832415 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.832596 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.832639 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:18:26.832720 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.829058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40475 (local address 127.0.98.2:51109)
0926 13:18:26.829115 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.829127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.829170 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.829351 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.829352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.829355 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:26.829434 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.829436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.830361 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.830362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.831634 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.831637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.832408 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.832409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.832504 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.832510 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.832581 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.832626 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":44}
I20260926 13:18:26.832731 17901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.829059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42371 (local address 127.0.98.2:36311)
0926 13:18:26.829342 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.829350 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:26.829402 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:26.829532 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:26.829533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:26.829534 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:26.829584 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.829585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.830449 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.830449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.831635 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:26.831637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.832408 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.832409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:26.832504 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.832510 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 13:18:26.832575 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 13:18:26.832626 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":56,"thread_start_us":164,"threads_started":1}
I20260926 13:18:26.833086 17564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293d07e7d8704c8ebcbf926c7fa5cd86" candidate_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722d3689e8ad4f559a2332f40d184c04" is_pre_election: true
I20260926 13:18:26.833132 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293d07e7d8704c8ebcbf926c7fa5cd86" candidate_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead88e6014da4f7ca058882a0aaa10af" is_pre_election: true
I20260926 13:18:26.833274 17564 raft_consensus.cc:2454] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6e1a4011cd4d3db056298f9eba39c8 in term 0.
I20260926 13:18:26.833292 17832 raft_consensus.cc:2454] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad6e1a4011cd4d3db056298f9eba39c8 in term 0.
I20260926 13:18:26.833479 17634 leader_election.cc:302] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722d3689e8ad4f559a2332f40d184c04, ad6e1a4011cd4d3db056298f9eba39c8; no voters:
I20260926 13:18:26.833624 17898 raft_consensus.cc:2781] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:18:26.833686 17898 raft_consensus.cc:482] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:18:26.833704 17898 raft_consensus.cc:3037] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:26.834379 17898 raft_consensus.cc:504] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.834511 17898 leader_election.cc:288] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [CANDIDATE]: Term 1 election: Requested vote from peers 722d3689e8ad4f559a2332f40d184c04 (127.0.98.1:42371), ead88e6014da4f7ca058882a0aaa10af (127.0.98.3:40475)
I20260926 13:18:26.834640 17564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293d07e7d8704c8ebcbf926c7fa5cd86" candidate_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "722d3689e8ad4f559a2332f40d184c04"
I20260926 13:18:26.834650 17832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "293d07e7d8704c8ebcbf926c7fa5cd86" candidate_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead88e6014da4f7ca058882a0aaa10af"
I20260926 13:18:26.834728 17832 raft_consensus.cc:3037] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:26.834728 17564 raft_consensus.cc:3037] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:26.835606 17832 raft_consensus.cc:2454] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6e1a4011cd4d3db056298f9eba39c8 in term 1.
I20260926 13:18:26.835616 17564 raft_consensus.cc:2454] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad6e1a4011cd4d3db056298f9eba39c8 in term 1.
I20260926 13:18:26.835762 17634 leader_election.cc:302] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 722d3689e8ad4f559a2332f40d184c04, ad6e1a4011cd4d3db056298f9eba39c8; no voters:
I20260926 13:18:26.835872 17898 raft_consensus.cc:2781] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:26.836035 17898 raft_consensus.cc:686] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 1 LEADER]: Becoming Leader. State: Replica: ad6e1a4011cd4d3db056298f9eba39c8, State: Running, Role: LEADER
I20260926 13:18:26.836144 17898 consensus_queue.cc:237] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } }
I20260926 13:18:26.836989 17418 catalog_manager.cc:5495] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad6e1a4011cd4d3db056298f9eba39c8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6e1a4011cd4d3db056298f9eba39c8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 } health_report { overall_health: UNKNOWN } } }
I20260926 13:18:26.843917 17889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.841830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.0.98.2:34525)
0926 13:18:26.841883 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:26.841885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:26.841894 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:18:26.841928 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:26.841929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:26.841943 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:26.842000 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:18:26.842090 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.842582 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.843286 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:26.843457 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:26.843505 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:26.843630 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:18:26.843719 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:18:26.843777 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:26.843788 (+ 11us) server_negotiation.cc:299] Negotiation successful
0926 13:18:26.843803 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:18:26.845536 17832 raft_consensus.cc:1261] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 1 FOLLOWER]: Refusing update from remote peer ad6e1a4011cd4d3db056298f9eba39c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:26.845556 17564 raft_consensus.cc:1261] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 1 FOLLOWER]: Refusing update from remote peer ad6e1a4011cd4d3db056298f9eba39c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:26.845882 17902 consensus_queue.cc:1035] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "722d3689e8ad4f559a2332f40d184c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:26.845990 17898 consensus_queue.cc:1035] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead88e6014da4f7ca058882a0aaa10af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:26.848984 17907 mvcc.cc:205] Tried to move back new op lower bound from 7333595983236816896 to 7333595983201533952. Current Snapshot: MvccSnapshot[applied={T|T < 7333595983236816896 or (T in {7333595983236816896})}]
I20260926 13:18:26.849046 17906 mvcc.cc:205] Tried to move back new op lower bound from 7333595983236816896 to 7333595983201533952. Current Snapshot: MvccSnapshot[applied={T|T < 7333595983236816896}]
I20260926 13:18:26.851191 17418 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:26.851269 17418 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:26.851987 17418 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5af8145f3512420891a7c80b7a6b46ed: Sending DeleteTablet for 3 replicas of tablet 293d07e7d8704c8ebcbf926c7fa5cd86
I20260926 13:18:26.852351 17678 tablet_service.cc:1514] Processing DeleteTablet for tablet 293d07e7d8704c8ebcbf926c7fa5cd86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60414
I20260926 13:18:26.852349 17544 tablet_service.cc:1514] Processing DeleteTablet for tablet 293d07e7d8704c8ebcbf926c7fa5cd86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53284
I20260926 13:18:26.852598 17916 tablet_replica.cc:330] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04: stopping tablet replica
I20260926 13:18:26.852599 17812 tablet_service.cc:1514] Processing DeleteTablet for tablet 293d07e7d8704c8ebcbf926c7fa5cd86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37502
I20260926 13:18:26.852754 17917 tablet_replica.cc:330] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: stopping tablet replica
I20260926 13:18:26.852852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17481
I20260926 13:18:26.852973 17916 raft_consensus.cc:2229] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:26.853039 17917 raft_consensus.cc:2229] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:18:26.853178 17916 raft_consensus.cc:2258] T 293d07e7d8704c8ebcbf926c7fa5cd86 P 722d3689e8ad4f559a2332f40d184c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:26.853520 17917 raft_consensus.cc:2258] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:26.853789 17918 tablet_replica.cc:330] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: stopping tablet replica
I20260926 13:18:26.853896 17918 raft_consensus.cc:2229] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:26.854018 17918 raft_consensus.cc:2258] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:26.854489 17918 ts_tablet_manager.cc:1905] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:26.855736 17917 ts_tablet_manager.cc:1905] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:26.856129 17918 ts_tablet_manager.cc:1918] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:26.856272 17918 log.cc:1196] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/293d07e7d8704c8ebcbf926c7fa5cd86
I20260926 13:18:26.856596 17918 ts_tablet_manager.cc:1939] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ead88e6014da4f7ca058882a0aaa10af: Deleting consensus metadata
I20260926 13:18:26.857064 17403 catalog_manager.cc:4841] TS ead88e6014da4f7ca058882a0aaa10af (127.0.98.3:40475): tablet 293d07e7d8704c8ebcbf926c7fa5cd86 (table test-table [id=2cb5521e519f43a989be4939965cae4d]) successfully deleted
I20260926 13:18:26.857448 17917 ts_tablet_manager.cc:1918] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:26.857540 17917 log.cc:1196] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/293d07e7d8704c8ebcbf926c7fa5cd86
I20260926 13:18:26.857779 17917 ts_tablet_manager.cc:1939] T 293d07e7d8704c8ebcbf926c7fa5cd86 P ad6e1a4011cd4d3db056298f9eba39c8: Deleting consensus metadata
I20260926 13:18:26.858126 17403 catalog_manager.cc:4841] TS ad6e1a4011cd4d3db056298f9eba39c8 (127.0.98.2:34525): tablet 293d07e7d8704c8ebcbf926c7fa5cd86 (table test-table [id=2cb5521e519f43a989be4939965cae4d]) successfully deleted
W20260926 13:18:26.859536 17406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42371 (error 108)
I20260926 13:18:26.859767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17615
I20260926 13:18:26.860404 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:26.859856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42371 (local address 127.0.0.1:53290)
0926 13:18:26.860330 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:26.860365 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456}
W20260926 13:18:26.860531 17406 catalog_manager.cc:4584] TS 722d3689e8ad4f559a2332f40d184c04 (127.0.98.1:42371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 293d07e7d8704c8ebcbf926c7fa5cd86: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42371: connect: Connection refused (error 111)
I20260926 13:18:26.863894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17749
I20260926 13:18:26.868326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17387
2026-09-26T13:18:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1883 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T13:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:26Z Disabled control of system clock
I20260926 13:18:26.895396 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36455
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:26.978030 17923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:26.978302 17923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:26.978325 17923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:26.979416 17923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:26.979449 17923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:26.979461 17923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:26.979475 17923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:26.979486 17923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:26.980721 17923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:26.980901 17923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:26.981132 17923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:26.985738 17928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.985852 17923 server_base.cc:1034] running on GCE node
W20260926 13:18:26.985734 17931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:26.985765 17929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:26.986155 17923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:26.986517 17923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:26.987663 17923 hybrid_clock.cc:648] HybridClock initialized: now 1790428706987657 us; error 33 us; skew 500 ppm
I20260926 13:18:26.988903 17923 webserver.cc:460] Webserver started at http://127.0.98.62:42195/ using document root <none> and password file <none>
I20260926 13:18:26.989166 17923 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:26.989214 17923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:26.989387 17923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:26.990483 17923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "91534de88b3648b19f579d0e458fc100"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "3517bed483f129e087d2146762ff71bc"
server_key_iv: "60d614eb128cb9344a88f8c24b63c00b"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.990865 17923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "91534de88b3648b19f579d0e458fc100"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "3517bed483f129e087d2146762ff71bc"
server_key_iv: "60d614eb128cb9344a88f8c24b63c00b"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.992103 17923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:18:26.992774 17937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:26.992931 17923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:26.993019 17923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "91534de88b3648b19f579d0e458fc100"
format_stamp: "Formatted at 2026-09-26 13:18:26 on dist-test-slave-f864"
server_key: "3517bed483f129e087d2146762ff71bc"
server_key_iv: "60d614eb128cb9344a88f8c24b63c00b"
server_key_version: "encryptionkey@0"
I20260926 13:18:26.993085 17923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:27.013057 17923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:27.013360 17923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:27.013510 17923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:27.017462 17923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36455
I20260926 13:18:27.017510 17989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36455 every 8 connection(s)
I20260926 13:18:27.017848 17923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:27.018826 17990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:27.021051 17990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100: Bootstrap starting.
I20260926 13:18:27.021629 17990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:27.021927 17990 log.cc:824] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:27.022639 17990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100: No bootstrap required, opened a new log
I20260926 13:18:27.022791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 17923
I20260926 13:18:27.022894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:27.023121 392 external_mini_cluster.cc:1428] Setting key 1f3d94fea9db03caadf83e4d48d55b96
I20260926 13:18:27.024647 17990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91534de88b3648b19f579d0e458fc100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36455 } }
I20260926 13:18:27.024799 17990 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:27.024848 17990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91534de88b3648b19f579d0e458fc100, State: Initialized, Role: FOLLOWER
I20260926 13:18:27.024972 17990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91534de88b3648b19f579d0e458fc100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36455 } }
I20260926 13:18:27.025051 17990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:27.025089 17990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:27.025135 17990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:27.025885 17990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91534de88b3648b19f579d0e458fc100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36455 } }
I20260926 13:18:27.025993 17990 leader_election.cc:302] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91534de88b3648b19f579d0e458fc100; no voters:
I20260926 13:18:27.026140 17990 leader_election.cc:288] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:27.026218 17995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:27.026364 17995 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [term 1 LEADER]: Becoming Leader. State: Replica: 91534de88b3648b19f579d0e458fc100, State: Running, Role: LEADER
I20260926 13:18:27.026393 17990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:27.026453 17995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91534de88b3648b19f579d0e458fc100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36455 } }
I20260926 13:18:27.026813 17995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91534de88b3648b19f579d0e458fc100. Latest consensus state: current_term: 1 leader_uuid: "91534de88b3648b19f579d0e458fc100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91534de88b3648b19f579d0e458fc100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36455 } } }
I20260926 13:18:27.026906 17995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:27.026827 17996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91534de88b3648b19f579d0e458fc100" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91534de88b3648b19f579d0e458fc100" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36455 } } }
I20260926 13:18:27.027019 17996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:27.027374 18001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:27.027885 18001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:27.030032 18001 catalog_manager.cc:1347] Generated new cluster ID: 26626ed1372c4eaa9d65c682acba088f
I20260926 13:18:27.030093 18001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:27.030560 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.023637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.0.98.62:36455)
0926 13:18:27.023828 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.023835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.023859 (+ 24us) server_negotiation.cc:407] Connection header received
0926 13:18:27.024623 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.024630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.024715 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.024863 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:27.025245 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.027409 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.028464 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.028875 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.029006 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.029735 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:27.029739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:27.029739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:27.029940 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:27.029982 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:27.029998 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:27.030056 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.030299 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.030386 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260926 13:18:27.037084 18001 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:27.037642 18001 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:27.047999 18001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100: Generated new TSK 0
I20260926 13:18:27.048213 18001 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:27.052309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36455
--builtin_ntp_servers=127.0.98.20:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:27.132241 18014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:27.132452 18014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:27.132471 18014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:27.133551 18014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:27.133589 18014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:27.133618 18014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:27.135007 18014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:27.135237 18014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:27.135541 18014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:27.141212 18020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:27.141335 18022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:27.141225 18019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:27.141548 18014 server_base.cc:1034] running on GCE node
I20260926 13:18:27.141757 18014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:27.142009 18014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:27.143185 18014 hybrid_clock.cc:648] HybridClock initialized: now 1790428707143165 us; error 32 us; skew 500 ppm
I20260926 13:18:27.145602 18014 webserver.cc:460] Webserver started at http://127.0.98.1:34069/ using document root <none> and password file <none>
I20260926 13:18:27.145891 18014 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:27.145951 18014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:27.146078 18014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:27.147717 18014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "03b4aa017cd84dc1bde778402ae36ce1"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "d997e2c4a39df81bd2166dd722dbcdb0"
server_key_iv: "1abc2a27a85cfdeac63c506814e514b2"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.148154 18014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "03b4aa017cd84dc1bde778402ae36ce1"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "d997e2c4a39df81bd2166dd722dbcdb0"
server_key_iv: "1abc2a27a85cfdeac63c506814e514b2"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.149713 18014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:27.150566 18028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.150774 18014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:18:27.150857 18014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "03b4aa017cd84dc1bde778402ae36ce1"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "d997e2c4a39df81bd2166dd722dbcdb0"
server_key_iv: "1abc2a27a85cfdeac63c506814e514b2"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.150921 18014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:27.187860 18014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:27.188143 18014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:27.188244 18014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:27.188434 18014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:27.188824 18014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:27.188865 18014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.188889 18014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:27.188939 18014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.194689 18014 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34751
I20260926 13:18:27.194739 18141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34751 every 8 connection(s)
I20260926 13:18:27.195060 18014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:27.197537 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18014
I20260926 13:18:27.197654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:18:27.197856 392 external_mini_cluster.cc:1428] Setting key f3bdc8ee89b7d231f83c47fd08f1e79a
I20260926 13:18:27.198748 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36455
--builtin_ntp_servers=127.0.98.20:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:18:27.200619 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.195745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36401 (local address 127.0.98.62:36455)
0926 13:18:27.195813 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.195816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.196133 (+ 317us) server_negotiation.cc:407] Connection header received
0926 13:18:27.196720 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.196721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.196739 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.196792 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:27.197575 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.198186 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.199244 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.199437 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.199508 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.200331 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:27.200333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:27.200333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:27.200338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:27.200353 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:27.200357 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:27.200420 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.200511 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.200541 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:18:27.200839 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.195757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36455 (local address 127.0.98.1:36401)
0926 13:18:27.195967 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.196032 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.196590 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.196868 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.196873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.196883 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:27.197482 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.197490 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.198279 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.198280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.199149 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.199151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.199514 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.199515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.199651 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.200194 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:27.200194 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:27.200222 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:27.200243 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:27.200422 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:27.200428 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:27.200431 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.200513 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.200623 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260926 13:18:27.201282 18142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36455
I20260926 13:18:27.201383 18142 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:27.201570 18142 heartbeater.cc:511] Master 127.0.98.62:36455 requested a full tablet report, sending...
I20260926 13:18:27.202086 17954 ts_manager.cc:194] Registered new tserver with Master: 03b4aa017cd84dc1bde778402ae36ce1 (127.0.98.1:34751)
I20260926 13:18:27.202702 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36401
W20260926 13:18:27.279855 18145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:27.280090 18145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:27.280125 18145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:27.281257 18145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:27.281317 18145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:27.281358 18145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:27.282747 18145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:27.283005 18145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:27.283295 18145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:27.288348 18153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:27.288484 18150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:27.288520 18145 server_base.cc:1034] running on GCE node
W20260926 13:18:27.288362 18151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:27.288836 18145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:27.289073 18145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:27.290236 18145 hybrid_clock.cc:648] HybridClock initialized: now 1790428707290225 us; error 38 us; skew 500 ppm
I20260926 13:18:27.291471 18145 webserver.cc:460] Webserver started at http://127.0.98.2:34889/ using document root <none> and password file <none>
I20260926 13:18:27.291705 18145 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:27.291771 18145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:27.291905 18145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:27.292955 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "902bf18788624bbbb743e644a40a8c9f"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "7fc75cd598d2bb428979bd20e987793b"
server_key_iv: "8977d031947b616bdc16f1690ab3142e"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.293322 18145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "902bf18788624bbbb743e644a40a8c9f"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "7fc75cd598d2bb428979bd20e987793b"
server_key_iv: "8977d031947b616bdc16f1690ab3142e"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.294688 18145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:27.295400 18159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.295578 18145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:27.295678 18145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "902bf18788624bbbb743e644a40a8c9f"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "7fc75cd598d2bb428979bd20e987793b"
server_key_iv: "8977d031947b616bdc16f1690ab3142e"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.295763 18145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:27.304720 18145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:27.304991 18145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:27.305132 18145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:27.305336 18145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:27.305660 18145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:27.305711 18145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.305743 18145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:27.305770 18145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.311357 18145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43463
I20260926 13:18:27.311404 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43463 every 8 connection(s)
I20260926 13:18:27.311721 18145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:18:27.313583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18145
I20260926 13:18:27.313694 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:18:27.313886 392 external_mini_cluster.cc:1428] Setting key 55ed76ffb2f89168a353970ac3ad5311
I20260926 13:18:27.314818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36455
--builtin_ntp_servers=127.0.98.20:36015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:18:27.317335 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.312358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47767 (local address 127.0.98.62:36455)
0926 13:18:27.312425 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.312427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.312669 (+ 242us) server_negotiation.cc:407] Connection header received
0926 13:18:27.313197 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.313198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.313215 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.313293 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:27.314021 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.314755 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.315771 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.316067 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.316129 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.317036 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:27.317037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:27.317038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:27.317042 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:27.317059 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:27.317063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:27.317125 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.317193 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.317220 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:18:27.317448 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.312360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36455 (local address 127.0.98.2:47767)
0926 13:18:27.312550 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.312582 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.313092 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.313348 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.313351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.313359 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:27.313926 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.313931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.314838 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.314839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.315690 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.315692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.316167 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.316169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.316334 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.316871 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:27.316872 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:27.316907 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:27.316936 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:27.317132 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:27.317136 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:27.317138 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.317185 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.317262 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260926 13:18:27.317853 18273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36455
I20260926 13:18:27.317955 18273 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:27.318149 18273 heartbeater.cc:511] Master 127.0.98.62:36455 requested a full tablet report, sending...
I20260926 13:18:27.318544 17954 ts_manager.cc:194] Registered new tserver with Master: 902bf18788624bbbb743e644a40a8c9f (127.0.98.2:43463)
I20260926 13:18:27.318869 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47767
W20260926 13:18:27.395152 18276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:27.395366 18276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:27.395403 18276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:27.396579 18276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:27.396641 18276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:27.396682 18276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:27.397993 18276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36455
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:27.398289 18276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:27.398581 18276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:27.403582 18282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:27.403628 18281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:27.403620 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:27.403697 18276 server_base.cc:1034] running on GCE node
I20260926 13:18:27.403892 18276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:27.404126 18276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:27.405258 18276 hybrid_clock.cc:648] HybridClock initialized: now 1790428707405247 us; error 31 us; skew 500 ppm
I20260926 13:18:27.406495 18276 webserver.cc:460] Webserver started at http://127.0.98.3:41727/ using document root <none> and password file <none>
I20260926 13:18:27.406742 18276 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:27.406801 18276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:27.406950 18276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:27.407995 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ae9c54d4f0c466f891f906234c557ea"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "21a517fff3586a461f7710a860021f60"
server_key_iv: "5194c0a71622feabe0e5e16bd3025ea8"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.408351 18276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae9c54d4f0c466f891f906234c557ea"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "21a517fff3586a461f7710a860021f60"
server_key_iv: "5194c0a71622feabe0e5e16bd3025ea8"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.409566 18276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:27.410251 18290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.410411 18276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:27.410502 18276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "8ae9c54d4f0c466f891f906234c557ea"
format_stamp: "Formatted at 2026-09-26 13:18:27 on dist-test-slave-f864"
server_key: "21a517fff3586a461f7710a860021f60"
server_key_iv: "5194c0a71622feabe0e5e16bd3025ea8"
server_key_version: "encryptionkey@0"
I20260926 13:18:27.410589 18276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:27.418619 18276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:27.418872 18276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:27.419001 18276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:27.419205 18276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:27.419507 18276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:27.419554 18276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.419592 18276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:27.419622 18276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:27.425014 18276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45321
I20260926 13:18:27.425065 18403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45321 every 8 connection(s)
I20260926 13:18:27.425410 18276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:18:27.430073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18276
I20260926 13:18:27.430254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:18:27.430471 392 external_mini_cluster.cc:1428] Setting key 0b8f3dd5d972406c355d3a824a28354a
I20260926 13:18:27.431363 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.426109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37905 (local address 127.0.98.62:36455)
0926 13:18:27.426190 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.426193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.426481 (+ 288us) server_negotiation.cc:407] Connection header received
0926 13:18:27.427034 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.427035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.427046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.427119 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:27.427828 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.428549 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.429755 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.430011 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.430087 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.431065 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:27.431066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:27.431067 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:27.431072 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:27.431087 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:27.431091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:27.431148 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.431236 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.431266 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:18:27.431584 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.426122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36455 (local address 127.0.98.3:37905)
0926 13:18:27.426336 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.426388 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.426915 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.427181 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.427185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.427193 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:27.427741 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.427746 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.428653 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.428654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.429687 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.429689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.430096 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.430097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.430269 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.430912 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:27.430913 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:27.430951 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:27.430979 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:27.431151 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:27.431155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:27.431158 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.431233 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.431343 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
I20260926 13:18:27.432008 18404 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36455
I20260926 13:18:27.432106 18404 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:27.432297 18404 heartbeater.cc:511] Master 127.0.98.62:36455 requested a full tablet report, sending...
I20260926 13:18:27.432627 17954 ts_manager.cc:194] Registered new tserver with Master: 8ae9c54d4f0c466f891f906234c557ea (127.0.98.3:45321)
I20260926 13:18:27.432998 17954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37905
I20260926 13:18:27.442103 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:18:27.446300 17994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.443434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.0.98.62:36455)
0926 13:18:27.443496 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.443498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.443607 (+ 109us) server_negotiation.cc:407] Connection header received
0926 13:18:27.443684 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.443685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.443697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.443742 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:27.443898 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.444366 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.445093 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.445297 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.445366 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.446059 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:27.446060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:27.446061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:27.446065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:27.446077 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:27.446081 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:27.446129 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.446194 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.446217 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:18:27.447809 17954 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:18:27.449328 17954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:18:27.455564 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.451574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.0.98.3:45321)
0926 13:18:27.452088 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.452093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.452102 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:18:27.452215 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.452217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.452233 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.452304 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:27.452485 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.453164 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.454473 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.455179 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.455258 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.455348 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.455448 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.455484 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":105,"threads_started":1}
I20260926 13:18:27.455561 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.451468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45321 (local address 127.0.0.1:41910)
0926 13:18:27.451981 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.451993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.452113 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.452360 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.452361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.452363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:27.452408 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.452409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.453257 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.453258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.454393 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.454396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.455270 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.455271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.455348 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.455362 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.455433 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.455484 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":86,"threads_started":1}
I20260926 13:18:27.456043 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.451462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34751 (local address 127.0.0.1:59768)
0926 13:18:27.451847 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.451919 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.452008 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.452327 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.452330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.452332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:27.452399 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.452401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.453693 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.453694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.454757 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.454759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.455813 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.455814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.455871 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.455875 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.455917 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.455953 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":28,"thread_start_us":58,"threads_started":1}
I20260926 13:18:27.456238 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.451671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43463 (local address 127.0.0.1:38512)
0926 13:18:27.452155 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.452164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.452215 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.452404 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.452405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.452407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:27.452444 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.452446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.453684 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.453685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.454775 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.454777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.455911 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.455912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.455962 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.455964 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.456015 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.456044 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260926 13:18:27.456238 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.451859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.98.2:43463)
0926 13:18:27.452052 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.452057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.452272 (+ 215us) server_negotiation.cc:407] Connection header received
0926 13:18:27.452315 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.452316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.452331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.452408 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:27.452502 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.453594 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.454871 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.455602 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.455791 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.455881 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.456035 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.456081 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260926 13:18:27.456442 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.451725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.0.98.1:34751)
0926 13:18:27.451916 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.451920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.452103 (+ 183us) server_negotiation.cc:407] Connection header received
0926 13:18:27.452152 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.452154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.452173 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.452247 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:27.452478 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.453604 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.454844 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.455609 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.455686 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.456262 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.456289 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.456343 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260926 13:18:27.456328 18338 tablet_service.cc:1467] Processing CreateTablet for tablet 09139f0cde0640bfaa9e0af9c4277e62 (DEFAULT_TABLE table=test-table [id=5453ac68e7984765b8f6e26001a4c9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:27.456683 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09139f0cde0640bfaa9e0af9c4277e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:27.456944 18207 tablet_service.cc:1467] Processing CreateTablet for tablet 09139f0cde0640bfaa9e0af9c4277e62 (DEFAULT_TABLE table=test-table [id=5453ac68e7984765b8f6e26001a4c9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:27.457091 18076 tablet_service.cc:1467] Processing CreateTablet for tablet 09139f0cde0640bfaa9e0af9c4277e62 (DEFAULT_TABLE table=test-table [id=5453ac68e7984765b8f6e26001a4c9e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:27.457258 18207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09139f0cde0640bfaa9e0af9c4277e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:27.457403 18076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09139f0cde0640bfaa9e0af9c4277e62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:27.459770 18418 tablet_bootstrap.cc:492] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Bootstrap starting.
I20260926 13:18:27.460414 18418 tablet_bootstrap.cc:654] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:27.460508 18420 tablet_bootstrap.cc:492] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: Bootstrap starting.
I20260926 13:18:27.460728 18418 log.cc:824] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:27.460907 18419 tablet_bootstrap.cc:492] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Bootstrap starting.
I20260926 13:18:27.461150 18420 tablet_bootstrap.cc:654] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:27.461424 18418 tablet_bootstrap.cc:492] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: No bootstrap required, opened a new log
I20260926 13:18:27.461499 18418 ts_tablet_manager.cc:1397] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:27.461483 18420 log.cc:824] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:27.461674 18419 tablet_bootstrap.cc:654] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:27.462040 18419 log.cc:824] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:27.462307 18420 tablet_bootstrap.cc:492] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: No bootstrap required, opened a new log
I20260926 13:18:27.462393 18420 ts_tablet_manager.cc:1397] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:27.462828 18419 tablet_bootstrap.cc:492] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: No bootstrap required, opened a new log
I20260926 13:18:27.462911 18419 ts_tablet_manager.cc:1397] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:27.463640 18418 raft_consensus.cc:348] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.463786 18418 raft_consensus.cc:374] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:27.463828 18418 raft_consensus.cc:729] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 902bf18788624bbbb743e644a40a8c9f, State: Initialized, Role: FOLLOWER
I20260926 13:18:27.463938 18418 consensus_queue.cc:260] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.464207 18418 ts_tablet_manager.cc:1428] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:27.464239 18273 heartbeater.cc:503] Master 127.0.98.62:36455 was elected leader, sending a full tablet report...
I20260926 13:18:27.465351 18420 raft_consensus.cc:348] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.465351 18419 raft_consensus.cc:348] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.465510 18420 raft_consensus.cc:374] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:27.465510 18419 raft_consensus.cc:374] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:27.465547 18419 raft_consensus.cc:729] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae9c54d4f0c466f891f906234c557ea, State: Initialized, Role: FOLLOWER
I20260926 13:18:27.465547 18420 raft_consensus.cc:729] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b4aa017cd84dc1bde778402ae36ce1, State: Initialized, Role: FOLLOWER
I20260926 13:18:27.465669 18419 consensus_queue.cc:260] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.465669 18420 consensus_queue.cc:260] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.465922 18419 ts_tablet_manager.cc:1428] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:27.465922 18420 ts_tablet_manager.cc:1428] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:27.465984 18404 heartbeater.cc:503] Master 127.0.98.62:36455 was elected leader, sending a full tablet report...
I20260926 13:18:27.466234 18142 heartbeater.cc:503] Master 127.0.98.62:36455 was elected leader, sending a full tablet report...
W20260926 13:18:27.562287 18274 tablet.cc:2369] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:18:27.581238 18426 raft_consensus.cc:482] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:27.581390 18426 raft_consensus.cc:504] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.581754 18426 leader_election.cc:288] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ae9c54d4f0c466f891f906234c557ea (127.0.98.3:45321), 902bf18788624bbbb743e644a40a8c9f (127.0.98.2:43463)
I20260926 13:18:27.585505 18417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.582273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48599 (local address 127.0.98.2:43463)
0926 13:18:27.582330 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.582332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.582340 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:18:27.582437 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.582438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.582446 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.582504 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:27.582653 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.583336 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.584593 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.585149 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.585220 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.585229 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.585396 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.585434 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 13:18:27.585505 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.582043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42665 (local address 127.0.98.3:45321)
0926 13:18:27.582101 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.582103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.582146 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:18:27.582207 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.582208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.582218 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.582271 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:27.582443 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.583205 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.584598 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.585145 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.585220 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.585229 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.585396 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.585435 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:18:27.585533 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.582106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43463 (local address 127.0.98.1:48599)
0926 13:18:27.582293 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.582302 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.582350 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.582526 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.582527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.582528 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:27.582579 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.582580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.583410 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.583411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.584501 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.584503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.585235 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.585236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.585322 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.585326 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.585380 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.585430 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":13,"thread_start_us":124,"threads_started":1}
I20260926 13:18:27.585532 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.581974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45321 (local address 127.0.98.1:42665)
0926 13:18:27.582037 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.582050 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:27.582097 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:27.582287 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:27.582288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:27.582290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:27.582366 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.582368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.583281 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.583282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.584501 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:27.584503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.585233 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.585234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:27.585312 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.585316 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:27.585380 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:18:27.585431 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":5}
I20260926 13:18:27.585923 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09139f0cde0640bfaa9e0af9c4277e62" candidate_uuid: "03b4aa017cd84dc1bde778402ae36ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902bf18788624bbbb743e644a40a8c9f" is_pre_election: true
I20260926 13:18:27.585981 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09139f0cde0640bfaa9e0af9c4277e62" candidate_uuid: "03b4aa017cd84dc1bde778402ae36ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae9c54d4f0c466f891f906234c557ea" is_pre_election: true
I20260926 13:18:27.586093 18358 raft_consensus.cc:2454] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b4aa017cd84dc1bde778402ae36ce1 in term 0.
I20260926 13:18:27.586102 18227 raft_consensus.cc:2454] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03b4aa017cd84dc1bde778402ae36ce1 in term 0.
I20260926 13:18:27.586342 18029 leader_election.cc:302] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b4aa017cd84dc1bde778402ae36ce1, 902bf18788624bbbb743e644a40a8c9f; no voters:
I20260926 13:18:27.586462 18426 raft_consensus.cc:2781] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:18:27.586519 18426 raft_consensus.cc:482] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:18:27.586544 18426 raft_consensus.cc:3037] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:27.587222 18426 raft_consensus.cc:504] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.587347 18426 leader_election.cc:288] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [CANDIDATE]: Term 1 election: Requested vote from peers 8ae9c54d4f0c466f891f906234c557ea (127.0.98.3:45321), 902bf18788624bbbb743e644a40a8c9f (127.0.98.2:43463)
I20260926 13:18:27.587512 18227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09139f0cde0640bfaa9e0af9c4277e62" candidate_uuid: "03b4aa017cd84dc1bde778402ae36ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "902bf18788624bbbb743e644a40a8c9f"
I20260926 13:18:27.587522 18358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09139f0cde0640bfaa9e0af9c4277e62" candidate_uuid: "03b4aa017cd84dc1bde778402ae36ce1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ae9c54d4f0c466f891f906234c557ea"
I20260926 13:18:27.587589 18227 raft_consensus.cc:3037] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:27.587597 18358 raft_consensus.cc:3037] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:27.588467 18358 raft_consensus.cc:2454] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b4aa017cd84dc1bde778402ae36ce1 in term 1.
I20260926 13:18:27.588467 18227 raft_consensus.cc:2454] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03b4aa017cd84dc1bde778402ae36ce1 in term 1.
I20260926 13:18:27.588618 18029 leader_election.cc:302] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b4aa017cd84dc1bde778402ae36ce1, 902bf18788624bbbb743e644a40a8c9f; no voters:
I20260926 13:18:27.588702 18426 raft_consensus.cc:2781] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:27.588845 18426 raft_consensus.cc:686] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 1 LEADER]: Becoming Leader. State: Replica: 03b4aa017cd84dc1bde778402ae36ce1, State: Running, Role: LEADER
I20260926 13:18:27.588940 18426 consensus_queue.cc:237] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } }
I20260926 13:18:27.589967 17954 catalog_manager.cc:5495] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 03b4aa017cd84dc1bde778402ae36ce1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "03b4aa017cd84dc1bde778402ae36ce1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b4aa017cd84dc1bde778402ae36ce1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 } health_report { overall_health: UNKNOWN } } }
I20260926 13:18:27.600131 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.597901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.0.98.1:34751)
0926 13:18:27.597936 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:27.597938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:27.597964 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:18:27.598041 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:27.598042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:27.598057 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:27.598120 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:18:27.598268 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.598743 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.599430 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:27.599641 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:27.599691 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:27.599803 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:18:27.599888 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:18:27.599941 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:27.600026 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 13:18:27.600041 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:18:27.601850 18358 raft_consensus.cc:1261] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 1 FOLLOWER]: Refusing update from remote peer 03b4aa017cd84dc1bde778402ae36ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:27.601886 18227 raft_consensus.cc:1261] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 1 FOLLOWER]: Refusing update from remote peer 03b4aa017cd84dc1bde778402ae36ce1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:27.602208 18426 consensus_queue.cc:1035] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae9c54d4f0c466f891f906234c557ea" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:27.602361 18428 consensus_queue.cc:1035] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "902bf18788624bbbb743e644a40a8c9f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:27.604841 18433 mvcc.cc:205] Tried to move back new op lower bound from 7333595986334535680 to 7333595986285330432. Current Snapshot: MvccSnapshot[applied={T|T < 7333595986334535680}]
I20260926 13:18:27.605481 18432 mvcc.cc:205] Tried to move back new op lower bound from 7333595986334535680 to 7333595986285330432. Current Snapshot: MvccSnapshot[applied={T|T < 7333595986334535680 or (T in {7333595986334535680})}]
I20260926 13:18:27.606880 17953 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:27.606974 17953 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:27.607712 17953 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91534de88b3648b19f579d0e458fc100: Sending DeleteTablet for 3 replicas of tablet 09139f0cde0640bfaa9e0af9c4277e62
I20260926 13:18:27.608052 18338 tablet_service.cc:1514] Processing DeleteTablet for tablet 09139f0cde0640bfaa9e0af9c4277e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:27 UTC) from {username='slave'} at 127.0.0.1:41910
I20260926 13:18:27.608062 18076 tablet_service.cc:1514] Processing DeleteTablet for tablet 09139f0cde0640bfaa9e0af9c4277e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:27 UTC) from {username='slave'} at 127.0.0.1:59768
I20260926 13:18:27.608273 18207 tablet_service.cc:1514] Processing DeleteTablet for tablet 09139f0cde0640bfaa9e0af9c4277e62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:27 UTC) from {username='slave'} at 127.0.0.1:38512
I20260926 13:18:27.608294 18442 tablet_replica.cc:330] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: stopping tablet replica
I20260926 13:18:27.608414 18442 raft_consensus.cc:2229] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:18:27.608502 18443 tablet_replica.cc:330] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: stopping tablet replica
I20260926 13:18:27.608518 18442 raft_consensus.cc:2258] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:27.608680 18441 tablet_replica.cc:330] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: stopping tablet replica
I20260926 13:18:27.608770 18441 raft_consensus.cc:2229] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:27.608878 18441 raft_consensus.cc:2258] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:27.608873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18014
I20260926 13:18:27.609138 18442 ts_tablet_manager.cc:1905] T 09139f0cde0640bfaa9e0af9c4277e62 P 03b4aa017cd84dc1bde778402ae36ce1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:27.609458 18443 raft_consensus.cc:2229] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:27.609588 18443 raft_consensus.cc:2258] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:27.609799 18441 ts_tablet_manager.cc:1905] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:27.610807 18443 ts_tablet_manager.cc:1905] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:27.611855 18441 ts_tablet_manager.cc:1918] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:27.611939 18441 log.cc:1196] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/09139f0cde0640bfaa9e0af9c4277e62
I20260926 13:18:27.612241 18441 ts_tablet_manager.cc:1939] T 09139f0cde0640bfaa9e0af9c4277e62 P 8ae9c54d4f0c466f891f906234c557ea: Deleting consensus metadata
I20260926 13:18:27.612437 18443 ts_tablet_manager.cc:1918] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:27.612514 18443 log.cc:1196] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/09139f0cde0640bfaa9e0af9c4277e62
I20260926 13:18:27.612686 17938 catalog_manager.cc:4841] TS 8ae9c54d4f0c466f891f906234c557ea (127.0.98.3:45321): tablet 09139f0cde0640bfaa9e0af9c4277e62 (table test-table [id=5453ac68e7984765b8f6e26001a4c9e2]) successfully deleted
I20260926 13:18:27.612798 18443 ts_tablet_manager.cc:1939] T 09139f0cde0640bfaa9e0af9c4277e62 P 902bf18788624bbbb743e644a40a8c9f: Deleting consensus metadata
I20260926 13:18:27.613260 17938 catalog_manager.cc:4841] TS 902bf18788624bbbb743e644a40a8c9f (127.0.98.2:43463): tablet 09139f0cde0640bfaa9e0af9c4277e62 (table test-table [id=5453ac68e7984765b8f6e26001a4c9e2]) successfully deleted
W20260926 13:18:27.615470 17941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34751 (error 108)
I20260926 13:18:27.615635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18145
I20260926 13:18:27.616266 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:27.615763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34751 (local address 127.0.0.1:59784)
0926 13:18:27.616207 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:27.616240 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
W20260926 13:18:27.616400 17941 catalog_manager.cc:4584] TS 03b4aa017cd84dc1bde778402ae36ce1 (127.0.98.1:34751): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09139f0cde0640bfaa9e0af9c4277e62: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34751: connect: Connection refused (error 111)
I20260926 13:18:27.620667 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18276
I20260926 13:18:27.625048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 17923
2026-09-26T13:18:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:27 dist-test-slave-f864 krb5kdc[18447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:27 dist-test-slave-f864 krb5kdc[18447](info): set up 2 sockets
Sep 26 13:18:27 dist-test-slave-f864 krb5kdc[18447](info): commencing operation
krb5kdc: starting...
W20260926 13:18:28.653875 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:28 dist-test-slave-f864 krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:28.766479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44695
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44695
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:28.847261 18460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:28.847468 18460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:28.847491 18460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:28.848662 18460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:28.848703 18460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:28.848717 18460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:28.848765 18460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:28.848783 18460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:28.850236 18460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:28.850483 18460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:28.850708 18460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:28.855482 18465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:28.855583 18460 server_base.cc:1034] running on GCE node
W20260926 13:18:28.855515 18466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:28.855525 18468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:28.855957 18460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:28.856284 18460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:28.857436 18460 hybrid_clock.cc:648] HybridClock initialized: now 1790428708857424 us; error 42 us; skew 500 ppm
Sep 26 13:18:28 dist-test-slave-f864 krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428708, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:28.859881 18460 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:28.860252 18460 webserver.cc:460] Webserver started at http://127.0.98.62:35013/ using document root <none> and password file <none>
I20260926 13:18:28.860423 18460 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:28.860462 18460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:28.860617 18460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:28.861608 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "01e0632ab12d4eb7aef3bd164003f2d0"
format_stamp: "Formatted at 2026-09-26 13:18:28 on dist-test-slave-f864"
server_key: "513025b4e788bce8bf8f9cbfbb16ab57"
server_key_iv: "bcf140011b8b27045065a187db5eb2c8"
server_key_version: "encryptionkey@0"
I20260926 13:18:28.861924 18460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "01e0632ab12d4eb7aef3bd164003f2d0"
format_stamp: "Formatted at 2026-09-26 13:18:28 on dist-test-slave-f864"
server_key: "513025b4e788bce8bf8f9cbfbb16ab57"
server_key_iv: "bcf140011b8b27045065a187db5eb2c8"
server_key_version: "encryptionkey@0"
I20260926 13:18:28.863207 18460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:28.863842 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:28.863986 18460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:28.864075 18460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "01e0632ab12d4eb7aef3bd164003f2d0"
format_stamp: "Formatted at 2026-09-26 13:18:28 on dist-test-slave-f864"
server_key: "513025b4e788bce8bf8f9cbfbb16ab57"
server_key_iv: "bcf140011b8b27045065a187db5eb2c8"
server_key_version: "encryptionkey@0"
I20260926 13:18:28.864128 18460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:28.899770 18460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:28.902153 18460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:28.902293 18460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:28.905929 18460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44695
I20260926 13:18:28.906010 18527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44695 every 8 connection(s)
I20260926 13:18:28.906379 18460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:28.907274 18528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:28.909534 18528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0: Bootstrap starting.
I20260926 13:18:28.910115 18528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:28.910439 18528 log.cc:824] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:28.911131 18528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0: No bootstrap required, opened a new log
I20260926 13:18:28.911303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18460
I20260926 13:18:28.911415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:28.911612 392 external_mini_cluster.cc:1428] Setting key 7b1a0f9ecda296c295a5b695913c817d
I20260926 13:18:28.913331 18528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44695 } }
I20260926 13:18:28.913519 18528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:28.913569 18528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01e0632ab12d4eb7aef3bd164003f2d0, State: Initialized, Role: FOLLOWER
I20260926 13:18:28.913718 18528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44695 } }
I20260926 13:18:28.913800 18528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:28.913838 18528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:28.913888 18528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:28.915020 18528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44695 } }
I20260926 13:18:28.915127 18528 leader_election.cc:302] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01e0632ab12d4eb7aef3bd164003f2d0; no voters:
I20260926 13:18:28.915277 18528 leader_election.cc:288] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:28.915326 18533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:28.915470 18533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [term 1 LEADER]: Becoming Leader. State: Replica: 01e0632ab12d4eb7aef3bd164003f2d0, State: Running, Role: LEADER
I20260926 13:18:28.915514 18528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:28.915563 18533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44695 } }
I20260926 13:18:28.915892 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01e0632ab12d4eb7aef3bd164003f2d0. Latest consensus state: current_term: 1 leader_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44695 } } }
I20260926 13:18:28.915978 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:28.916235 18539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:28.916749 18539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:28.915908 18534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01e0632ab12d4eb7aef3bd164003f2d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44695 } } }
I20260926 13:18:28.917109 18534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:28.918720 18539 catalog_manager.cc:1347] Generated new cluster ID: 1ba88b4203f64b22903445dd8a6314db
I20260926 13:18:28.918769 18539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:28 dist-test-slave-f864 krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:28.924391 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:28.912097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.0.98.62:44695)
0926 13:18:28.912275 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:28.912284 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:28.912318 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:18:28.913293 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:28.913302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:28.913378 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:28.913524 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:28.914349 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:28.915821 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:28.916955 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:28.917467 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:28.917583 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:28.921105 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:28.921110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:28.921110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:28.921127 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:28.922875 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:28.923240 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:28.923243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:28.923243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:28.923345 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:28.923550 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:28.923551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:28.923551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:28.923786 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:28.923954 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:28.924052 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:18:28.924201 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260926 13:18:28.925612 18539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:28.926349 18539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:28.935756 18539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0: Generated new TSK 0
I20260926 13:18:28.935940 18539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:18:28.961385 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44695
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:29.043110 18554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:29.043386 18554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:29.043426 18554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:29.044680 18554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:29.044739 18554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:29.044785 18554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:29.046372 18554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:29.046638 18554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:29.046859 18554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:29.051927 18562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.052006 18559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.052086 18560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:29.052248 18554 server_base.cc:1034] running on GCE node
I20260926 13:18:29.052448 18554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:29.052675 18554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:29.053822 18554 hybrid_clock.cc:648] HybridClock initialized: now 1790428709053809 us; error 32 us; skew 500 ppm
Sep 26 13:18:29 dist-test-slave-f864 krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:29.056305 18554 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:18:29.056754 18554 webserver.cc:460] Webserver started at http://127.0.98.1:45337/ using document root <none> and password file <none>
I20260926 13:18:29.056931 18554 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:29.056972 18554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:29.057061 18554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:29.058046 18554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "ee811b71d2014e769452d8dca35478ab"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "d6cb9fe12c32a1260fa147c0cabd6eba"
server_key_iv: "ddf6368332e3753a45bc97861809d8e4"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.058418 18554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ee811b71d2014e769452d8dca35478ab"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "d6cb9fe12c32a1260fa147c0cabd6eba"
server_key_iv: "ddf6368332e3753a45bc97861809d8e4"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.059614 18554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:29.060253 18569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.060393 18554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:29.060469 18554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "ee811b71d2014e769452d8dca35478ab"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "d6cb9fe12c32a1260fa147c0cabd6eba"
server_key_iv: "ddf6368332e3753a45bc97861809d8e4"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.060547 18554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:29.084476 18554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:29.086792 18554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:29.086954 18554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:29.087182 18554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:29.087489 18554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:29.087540 18554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.087584 18554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:29.087617 18554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.093248 18554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42971
I20260926 13:18:29.093322 18682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42971 every 8 connection(s)
I20260926 13:18:29.093637 18554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:29.096374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18554
I20260926 13:18:29.096480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:18:29.096676 392 external_mini_cluster.cc:1428] Setting key fce1b5cb06188b0c258b6deae0974490
Sep 26 13:18:29 dist-test-slave-f864 krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:29.103523 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.094368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48165 (local address 127.0.98.62:44695)
0926 13:18:29.094426 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.094428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.094612 (+ 184us) server_negotiation.cc:407] Connection header received
0926 13:18:29.095521 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.095522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.095542 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.095592 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:29.096151 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.096799 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.097884 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.098100 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.098184 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.100429 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:29.100431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:29.100432 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:29.100440 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:29.102089 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.102499 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.102500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.102501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.102564 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.102784 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.102785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.102785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.102926 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:29.103001 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.103303 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.103406 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:18:29.103602 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.094348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44695 (local address 127.0.98.1:48165)
0926 13:18:29.094501 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.094535 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.095438 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.095646 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.095649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.095887 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:29.096083 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.096087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.096882 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.096883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.097797 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.097800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.098197 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.098198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.098339 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.098805 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:29.098815 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:29.100341 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:29.102200 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:29.102207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:29.102209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:29.102421 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:29.102632 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:29.102633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:29.102633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:29.102724 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:29.103030 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:29.103038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:29.103158 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.103264 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.103411 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260926 13:18:29.104265 18683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44695
I20260926 13:18:29.104386 18683 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:29.104566 18683 heartbeater.cc:511] Master 127.0.98.62:44695 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:18:29.105106 18492 ts_manager.cc:194] Registered new tserver with Master: ee811b71d2014e769452d8dca35478ab (127.0.98.1:42971)
I20260926 13:18:29.105667 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:18:29.121562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44695
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:29.200336 18688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:29.200569 18688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:29.200610 18688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:29.201745 18688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:29.201808 18688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:29.201854 18688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:29.203353 18688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:29.203608 18688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:29.203819 18688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:29.209115 18694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.209162 18696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.209122 18693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:29.209239 18688 server_base.cc:1034] running on GCE node
I20260926 13:18:29.209586 18688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:29.209815 18688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:29.210968 18688 hybrid_clock.cc:648] HybridClock initialized: now 1790428709210960 us; error 34 us; skew 500 ppm
Sep 26 13:18:29 dist-test-slave-f864 krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:29.213378 18688 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:18:29.213749 18688 webserver.cc:460] Webserver started at http://127.0.98.2:44311/ using document root <none> and password file <none>
I20260926 13:18:29.213919 18688 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:29.213961 18688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:29.214048 18688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:29.215122 18688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "56b1fd6094f149148957afa6cc605a2c"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "dcefd38689cb2ea1ab5ebee39dc7a2bc"
server_key_iv: "b28dca86342871ecc7e6a7654925f9e9"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.215436 18688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "56b1fd6094f149148957afa6cc605a2c"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "dcefd38689cb2ea1ab5ebee39dc7a2bc"
server_key_iv: "b28dca86342871ecc7e6a7654925f9e9"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.216851 18688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:29.217608 18703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.217880 18688 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:29.217976 18688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "56b1fd6094f149148957afa6cc605a2c"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "dcefd38689cb2ea1ab5ebee39dc7a2bc"
server_key_iv: "b28dca86342871ecc7e6a7654925f9e9"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.218050 18688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:29.228605 18688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:29.230419 18688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:29.230561 18688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:29.230763 18688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:29.231047 18688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:29.231081 18688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.231118 18688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:29.231144 18688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.236650 18688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40777
I20260926 13:18:29.236727 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40777 every 8 connection(s)
I20260926 13:18:29.237001 18688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:18:29 dist-test-slave-f864 krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:29.245702 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.237672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55185 (local address 127.0.98.62:44695)
0926 13:18:29.237727 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.237729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.237914 (+ 185us) server_negotiation.cc:407] Connection header received
0926 13:18:29.238935 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.238937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.238950 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.238996 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:29.239602 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.240099 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.241083 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.241277 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.241350 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.243393 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:29.243394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:29.243395 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:29.243402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:29.244566 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.244949 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.244949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.244950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.244996 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.245191 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.245191 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.245191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.245284 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:29.245333 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.245565 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.245638 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 13:18:29.245754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18688
I20260926 13:18:29.245857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:18:29.245985 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.237652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44695 (local address 127.0.98.2:55185)
0926 13:18:29.237805 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.237836 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.238822 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.239083 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.239087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.239326 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:29.239501 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.239505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.240187 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.240188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.240973 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.240975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.241354 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.241355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.241475 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.241908 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:29.241916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:29.243308 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:29.244634 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:29.244639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:29.244640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:29.244889 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:29.245048 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:29.245048 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:29.245048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:29.245141 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:29.245335 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:29.245338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:29.245457 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.245534 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.245729 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260926 13:18:29.246095 392 external_mini_cluster.cc:1428] Setting key f6c5f9aca3e1048b817494c9b7ed8896
I20260926 13:18:29.246491 18817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44695
I20260926 13:18:29.246593 18817 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:29.246776 18817 heartbeater.cc:511] Master 127.0.98.62:44695 requested a full tablet report, sending...
I20260926 13:18:29.247169 18492 ts_manager.cc:194] Registered new tserver with Master: 56b1fd6094f149148957afa6cc605a2c (127.0.98.2:40777)
I20260926 13:18:29.247532 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55185
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:18:29.270360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44695
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:29.349314 18822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:29.349509 18822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:29.349530 18822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:29.350739 18822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:29.350780 18822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:29.350829 18822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:29.352509 18822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:29.352756 18822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:29.352964 18822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:29.357717 18827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.357720 18830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.357721 18828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:29.357995 18822 server_base.cc:1034] running on GCE node
I20260926 13:18:29.358318 18822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:29.358558 18822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:29.359712 18822 hybrid_clock.cc:648] HybridClock initialized: now 1790428709359700 us; error 40 us; skew 500 ppm
Sep 26 13:18:29 dist-test-slave-f864 krb5kdc[18447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:29.366771 18822 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:18:29.367244 18822 webserver.cc:460] Webserver started at http://127.0.98.3:46175/ using document root <none> and password file <none>
I20260926 13:18:29.367507 18822 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:29.367565 18822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:29.367700 18822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:29.368688 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "f93a02e0865f49d0b52ba62e69b483c6"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "b2b81674631a3b86dff39131a0975946"
server_key_iv: "b1f5f20e00a6a6239bbf19ea5cf4b80b"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.369045 18822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f93a02e0865f49d0b52ba62e69b483c6"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "b2b81674631a3b86dff39131a0975946"
server_key_iv: "b1f5f20e00a6a6239bbf19ea5cf4b80b"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.370371 18822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:29.371122 18837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.371269 18822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:29.371357 18822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "f93a02e0865f49d0b52ba62e69b483c6"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "b2b81674631a3b86dff39131a0975946"
server_key_iv: "b1f5f20e00a6a6239bbf19ea5cf4b80b"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.371444 18822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:29.379508 18822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:29.381162 18822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:29.381300 18822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:29.381490 18822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:29.381753 18822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:29.381800 18822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.381837 18822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:29.381871 18822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.387501 18822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35865
I20260926 13:18:29.387548 18950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35865 every 8 connection(s)
I20260926 13:18:29.387827 18822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 13:18:29 dist-test-slave-f864 krb5kdc[18447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:29.394995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18822
I20260926 13:18:29.395124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:18:29.395306 392 external_mini_cluster.cc:1428] Setting key 98923c5e493011acf5d9bb1b8abd736c
I20260926 13:18:29.396798 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.388419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34625 (local address 127.0.98.62:44695)
0926 13:18:29.388474 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.388476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.388656 (+ 180us) server_negotiation.cc:407] Connection header received
0926 13:18:29.389541 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.389542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.389550 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.389604 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:29.390134 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.390711 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.391713 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.391881 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.391928 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.393896 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:29.393897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:29.393898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:29.393904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:29.395417 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.396003 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.396004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.396004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.396066 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.396330 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.396331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.396331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.396431 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:29.396481 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.396677 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.396738 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:18:29.397042 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.388397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44695 (local address 127.0.98.3:34625)
0926 13:18:29.388550 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.388588 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.389457 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.389662 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.389665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.389902 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:29.390070 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.390073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.390805 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.390806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.391641 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.391643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.391973 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.391974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.392079 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.392509 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:29.392516 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:29.393820 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:29.395599 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:29.395605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:29.395607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:29.395925 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:29.396135 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:29.396136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:29.396136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:29.396271 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:29.396473 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:29.396476 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:29.396577 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.396651 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.396774 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260926 13:18:29.397445 18951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44695
I20260926 13:18:29.397538 18951 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:29.397703 18951 heartbeater.cc:511] Master 127.0.98.62:44695 requested a full tablet report, sending...
I20260926 13:18:29.398029 18492 ts_manager.cc:194] Registered new tserver with Master: f93a02e0865f49d0b52ba62e69b483c6 (127.0.98.3:35865)
I20260926 13:18:29.398396 18492 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34625
I20260926 13:18:29.406800 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:18:29.413930 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.408135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.0.98.62:44695)
0926 13:18:29.408181 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.408183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.408299 (+ 116us) server_negotiation.cc:407] Connection header received
0926 13:18:29.408353 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.408354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.408362 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.408407 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:29.408925 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.409379 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.410090 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.410335 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.410386 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.411634 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:29.411635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:29.411635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:29.411646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:29.412905 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.413291 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.413292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.413292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.413341 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:29.413536 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:29.413536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:29.413536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:29.413654 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:29.413734 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.413822 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.413868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:18:29.415604 18492 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:18:29.417476 18492 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:18:29.423851 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.419495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35865 (local address 127.0.0.1:53648)
0926 13:18:29.419819 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.419863 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.419977 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.420248 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.420249 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.420250 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:29.420313 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.420317 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.421647 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.421647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.422730 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.422732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.423537 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.423626 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.423640 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.423717 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.423770 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":68,"thread_start_us":63,"threads_started":1}
I20260926 13:18:29.423885 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.419596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.0.98.3:35865)
0926 13:18:29.419765 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.419770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.419963 (+ 193us) server_negotiation.cc:407] Connection header received
0926 13:18:29.420051 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.420054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.420069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.420163 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:29.420402 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.421486 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.422810 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.423451 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423525 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.423582 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.423755 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.423799 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260926 13:18:29.424046 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.419707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42971 (local address 127.0.0.1:33798)
0926 13:18:29.420138 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.420148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.420226 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.420553 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.420554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.420556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:29.420586 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.420587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.421602 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.421603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.423004 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.423007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.423819 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.423876 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.423880 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.423929 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.423970 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":84,"thread_start_us":57,"threads_started":1}
I20260926 13:18:29.424286 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.419525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40777 (local address 127.0.0.1:52662)
0926 13:18:29.419821 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.419861 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.419933 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.420240 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.420244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.420246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:29.420313 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.420316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.421597 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.421598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.423004 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.423007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.424065 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.424066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.424130 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.424134 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.424176 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.424213 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":113,"thread_start_us":65,"threads_started":1}
I20260926 13:18:29.424295 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.419902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.98.1:42971)
0926 13:18:29.420075 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.420080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.420201 (+ 121us) server_negotiation.cc:407] Connection header received
0926 13:18:29.420343 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.420346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.420362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.420427 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:29.420653 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.421442 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423113 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.423673 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423734 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.423787 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.424191 (+ 404us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.424233 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
I20260926 13:18:29.424902 18885 tablet_service.cc:1467] Processing CreateTablet for tablet 27f49b9a70e1450889647cd83b21e063 (DEFAULT_TABLE table=test-table [id=f3b09185acab409d93ddb9af5b0539ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:29.424952 18617 tablet_service.cc:1467] Processing CreateTablet for tablet 27f49b9a70e1450889647cd83b21e063 (DEFAULT_TABLE table=test-table [id=f3b09185acab409d93ddb9af5b0539ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:29.425216 18885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f49b9a70e1450889647cd83b21e063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:29.425241 18617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f49b9a70e1450889647cd83b21e063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:29.426402 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.419647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.98.2:40777)
0926 13:18:29.419838 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.419842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.419912 (+ 70us) server_negotiation.cc:407] Connection header received
0926 13:18:29.420052 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.420054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.420069 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.420142 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:29.420394 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.421502 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423089 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.423788 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.423850 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.423901 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.426274 (+ 2373us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.426324 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":75,"threads_started":1}
I20260926 13:18:29.427120 18751 tablet_service.cc:1467] Processing CreateTablet for tablet 27f49b9a70e1450889647cd83b21e063 (DEFAULT_TABLE table=test-table [id=f3b09185acab409d93ddb9af5b0539ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:29.427443 18751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27f49b9a70e1450889647cd83b21e063. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:29.428040 18965 tablet_bootstrap.cc:492] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Bootstrap starting.
I20260926 13:18:29.428349 18966 tablet_bootstrap.cc:492] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Bootstrap starting.
I20260926 13:18:29.428815 18965 tablet_bootstrap.cc:654] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:29.429013 18966 tablet_bootstrap.cc:654] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:29.429457 18965 log.cc:824] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:29.429577 18966 log.cc:824] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:29.430302 18965 tablet_bootstrap.cc:492] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: No bootstrap required, opened a new log
I20260926 13:18:29.430382 18965 ts_tablet_manager.cc:1397] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:29.430469 18966 tablet_bootstrap.cc:492] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: No bootstrap required, opened a new log
I20260926 13:18:29.430567 18966 ts_tablet_manager.cc:1397] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:29.430768 18969 tablet_bootstrap.cc:492] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Bootstrap starting.
I20260926 13:18:29.431376 18969 tablet_bootstrap.cc:654] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:29.431947 18969 log.cc:824] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:29.431988 18966 raft_consensus.cc:348] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.432195 18966 raft_consensus.cc:374] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:29.432243 18966 raft_consensus.cc:729] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93a02e0865f49d0b52ba62e69b483c6, State: Initialized, Role: FOLLOWER
I20260926 13:18:29.432364 18966 consensus_queue.cc:260] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.432641 18951 heartbeater.cc:503] Master 127.0.98.62:44695 was elected leader, sending a full tablet report...
I20260926 13:18:29.432535 18965 raft_consensus.cc:348] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.432643 18966 ts_tablet_manager.cc:1428] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:29.432699 18965 raft_consensus.cc:374] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:29.432753 18965 raft_consensus.cc:729] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee811b71d2014e769452d8dca35478ab, State: Initialized, Role: FOLLOWER
I20260926 13:18:29.432860 18969 tablet_bootstrap.cc:492] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: No bootstrap required, opened a new log
I20260926 13:18:29.432861 18965 consensus_queue.cc:260] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.432919 18969 ts_tablet_manager.cc:1397] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:29.433163 18683 heartbeater.cc:503] Master 127.0.98.62:44695 was elected leader, sending a full tablet report...
I20260926 13:18:29.433147 18965 ts_tablet_manager.cc:1428] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:29.434755 18969 raft_consensus.cc:348] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.434877 18969 raft_consensus.cc:374] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:29.434906 18969 raft_consensus.cc:729] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b1fd6094f149148957afa6cc605a2c, State: Initialized, Role: FOLLOWER
I20260926 13:18:29.435029 18969 consensus_queue.cc:260] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.435268 18969 ts_tablet_manager.cc:1428] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:18:29.435302 18817 heartbeater.cc:503] Master 127.0.98.62:44695 was elected leader, sending a full tablet report...
W20260926 13:18:29.487529 18818 tablet.cc:2369] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:18:29.594445 18684 tablet.cc:2369] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:18:29.637949 18972 raft_consensus.cc:482] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:29.638124 18972 raft_consensus.cc:504] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
W20260926 13:18:29.638348 18952 tablet.cc:2369] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:18:29.638586 18972 leader_election.cc:288] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93a02e0865f49d0b52ba62e69b483c6 (127.0.98.3:35865), 56b1fd6094f149148957afa6cc605a2c (127.0.98.2:40777)
I20260926 13:18:29.642616 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.638908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53913 (local address 127.0.98.2:40777)
0926 13:18:29.638954 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.638956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.639000 (+ 44us) server_negotiation.cc:407] Connection header received
0926 13:18:29.639090 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.639091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.639102 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.639164 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:29.639348 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.640128 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.641688 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.642264 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.642339 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.642349 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.642508 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.642547 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:18:29.642616 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.638877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60427 (local address 127.0.98.3:35865)
0926 13:18:29.638926 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.638929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.638980 (+ 51us) server_negotiation.cc:407] Connection header received
0926 13:18:29.639064 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.639065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.639074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.639159 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:29.639356 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.640132 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.641688 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.642263 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.642339 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.642349 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.642508 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.642547 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:18:29.642661 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.638785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35865 (local address 127.0.98.1:60427)
0926 13:18:29.638922 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.638934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.638999 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.639181 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.639182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.639184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:29.639284 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.639285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.640227 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.640228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.641598 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.641600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.642342 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.642343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.642428 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.642436 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.642493 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.642547 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":46,"thread_start_us":70,"threads_started":1}
I20260926 13:18:29.642856 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.638806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40777 (local address 127.0.98.1:53913)
0926 13:18:29.638937 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.638948 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:29.639011 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:29.639206 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:29.639207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:29.639208 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:29.639261 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.639263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.640232 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.640233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.641609 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:29.641611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.642341 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.642342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:29.642428 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.642435 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 13:18:29.642493 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 13:18:29.642547 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":115,"thread_start_us":58,"threads_started":1}
I20260926 13:18:29.643031 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f49b9a70e1450889647cd83b21e063" candidate_uuid: "ee811b71d2014e769452d8dca35478ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93a02e0865f49d0b52ba62e69b483c6" is_pre_election: true
I20260926 13:18:29.643199 18905 raft_consensus.cc:2454] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee811b71d2014e769452d8dca35478ab in term 0.
I20260926 13:18:29.643260 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f49b9a70e1450889647cd83b21e063" candidate_uuid: "ee811b71d2014e769452d8dca35478ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b1fd6094f149148957afa6cc605a2c" is_pre_election: true
I20260926 13:18:29.643402 18771 raft_consensus.cc:2454] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee811b71d2014e769452d8dca35478ab in term 0.
I20260926 13:18:29.643458 18571 leader_election.cc:302] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee811b71d2014e769452d8dca35478ab, f93a02e0865f49d0b52ba62e69b483c6; no voters:
I20260926 13:18:29.643603 18972 raft_consensus.cc:2781] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:18:29.643673 18972 raft_consensus.cc:482] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:18:29.643697 18972 raft_consensus.cc:3037] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:29.644384 18972 raft_consensus.cc:504] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.644508 18972 leader_election.cc:288] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [CANDIDATE]: Term 1 election: Requested vote from peers f93a02e0865f49d0b52ba62e69b483c6 (127.0.98.3:35865), 56b1fd6094f149148957afa6cc605a2c (127.0.98.2:40777)
I20260926 13:18:29.644618 18905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f49b9a70e1450889647cd83b21e063" candidate_uuid: "ee811b71d2014e769452d8dca35478ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93a02e0865f49d0b52ba62e69b483c6"
I20260926 13:18:29.644690 18905 raft_consensus.cc:3037] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:29.644666 18771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27f49b9a70e1450889647cd83b21e063" candidate_uuid: "ee811b71d2014e769452d8dca35478ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b1fd6094f149148957afa6cc605a2c"
I20260926 13:18:29.644739 18771 raft_consensus.cc:3037] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:29.645321 18905 raft_consensus.cc:2454] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee811b71d2014e769452d8dca35478ab in term 1.
I20260926 13:18:29.645390 18771 raft_consensus.cc:2454] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee811b71d2014e769452d8dca35478ab in term 1.
I20260926 13:18:29.645512 18571 leader_election.cc:302] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee811b71d2014e769452d8dca35478ab, f93a02e0865f49d0b52ba62e69b483c6; no voters:
I20260926 13:18:29.645632 18972 raft_consensus.cc:2781] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:29.645809 18972 raft_consensus.cc:686] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 1 LEADER]: Becoming Leader. State: Replica: ee811b71d2014e769452d8dca35478ab, State: Running, Role: LEADER
I20260926 13:18:29.645908 18972 consensus_queue.cc:237] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } }
I20260926 13:18:29.646811 18492 catalog_manager.cc:5495] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab reported cstate change: term changed from 0 to 1, leader changed from <none> to ee811b71d2014e769452d8dca35478ab (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ee811b71d2014e769452d8dca35478ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee811b71d2014e769452d8dca35478ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42971 } health_report { overall_health: HEALTHY } } }
I20260926 13:18:29.708062 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.705819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33800 (local address 127.0.98.1:42971)
0926 13:18:29.705878 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.705880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.705890 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:18:29.705927 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.705928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.705948 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.706011 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:18:29.706134 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.706633 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.707329 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.707575 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.707628 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.707745 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:18:29.707828 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:18:29.707912 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.707940 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.707953 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:18:29.709920 18771 raft_consensus.cc:1261] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 1 FOLLOWER]: Refusing update from remote peer ee811b71d2014e769452d8dca35478ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:29.709947 18905 raft_consensus.cc:1261] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 1 FOLLOWER]: Refusing update from remote peer ee811b71d2014e769452d8dca35478ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:29.710302 18976 consensus_queue.cc:1035] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "56b1fd6094f149148957afa6cc605a2c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:29.710438 18972 consensus_queue.cc:1035] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93a02e0865f49d0b52ba62e69b483c6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:29.712682 18980 mvcc.cc:205] Tried to move back new op lower bound from 7333595994969038848 to 7333595994710396928. Current Snapshot: MvccSnapshot[applied={T|T < 7333595994969038848}]
I20260926 13:18:29.712922 18979 mvcc.cc:205] Tried to move back new op lower bound from 7333595994969038848 to 7333595994710396928. Current Snapshot: MvccSnapshot[applied={T|T < 7333595994969038848 or (T in {7333595994969038848})}]
I20260926 13:18:29.713217 18981 mvcc.cc:205] Tried to move back new op lower bound from 7333595994969038848 to 7333595994710396928. Current Snapshot: MvccSnapshot[applied={T|T < 7333595994969038848}]
I20260926 13:18:29.714543 18492 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:29.714625 18492 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:29.715344 18492 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 01e0632ab12d4eb7aef3bd164003f2d0: Sending DeleteTablet for 3 replicas of tablet 27f49b9a70e1450889647cd83b21e063
I20260926 13:18:29.715683 18751 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f49b9a70e1450889647cd83b21e063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52662
I20260926 13:18:29.715752 18885 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f49b9a70e1450889647cd83b21e063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53648
I20260926 13:18:29.715824 18617 tablet_service.cc:1514] Processing DeleteTablet for tablet 27f49b9a70e1450889647cd83b21e063 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33798
I20260926 13:18:29.715953 18989 tablet_replica.cc:330] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: stopping tablet replica
I20260926 13:18:29.716048 18989 raft_consensus.cc:2229] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:29.716066 18991 tablet_replica.cc:330] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: stopping tablet replica
I20260926 13:18:29.716135 18989 raft_consensus.cc:2258] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:29.716152 18991 raft_consensus.cc:2229] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:18:29.716269 18991 raft_consensus.cc:2258] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:29.716517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18554
I20260926 13:18:29.716712 18991 ts_tablet_manager.cc:1905] T 27f49b9a70e1450889647cd83b21e063 P ee811b71d2014e769452d8dca35478ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:29.717144 18990 tablet_replica.cc:330] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: stopping tablet replica
I20260926 13:18:29.717226 18990 raft_consensus.cc:2229] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:29.717286 18990 raft_consensus.cc:2258] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:29.717684 18990 ts_tablet_manager.cc:1905] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:29.719769 18990 ts_tablet_manager.cc:1918] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:29.719826 18990 log.cc:1196] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/27f49b9a70e1450889647cd83b21e063
I20260926 13:18:29.720080 18990 ts_tablet_manager.cc:1939] T 27f49b9a70e1450889647cd83b21e063 P 56b1fd6094f149148957afa6cc605a2c: Deleting consensus metadata
I20260926 13:18:29.720491 18476 catalog_manager.cc:4841] TS 56b1fd6094f149148957afa6cc605a2c (127.0.98.2:40777): tablet 27f49b9a70e1450889647cd83b21e063 (table test-table [id=f3b09185acab409d93ddb9af5b0539ac]) successfully deleted
I20260926 13:18:29.722571 18989 ts_tablet_manager.cc:1905] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 13:18:29.722662 18476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42971 (error 108)
I20260926 13:18:29.722856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18688
I20260926 13:18:29.724148 18989 ts_tablet_manager.cc:1918] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:29.724249 18989 log.cc:1196] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/27f49b9a70e1450889647cd83b21e063
I20260926 13:18:29.724370 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.724233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42971 (local address 127.0.0.1:33810)
0926 13:18:29.724280 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:29.724314 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:18:29.724505 18476 catalog_manager.cc:4584] TS ee811b71d2014e769452d8dca35478ab (127.0.98.1:42971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27f49b9a70e1450889647cd83b21e063: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42971: connect: Connection refused (error 111)
I20260926 13:18:29.724581 18989 ts_tablet_manager.cc:1939] T 27f49b9a70e1450889647cd83b21e063 P f93a02e0865f49d0b52ba62e69b483c6: Deleting consensus metadata
I20260926 13:18:29.725001 18477 catalog_manager.cc:4841] TS f93a02e0865f49d0b52ba62e69b483c6 (127.0.98.3:35865): tablet 27f49b9a70e1450889647cd83b21e063 (table test-table [id=f3b09185acab409d93ddb9af5b0539ac]) successfully deleted
I20260926 13:18:29.728752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18822
I20260926 13:18:29.733413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18460
2026-09-26T13:18:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2108 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T13:18:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:29Z Disabled control of system clock
I20260926 13:18:29.760329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45979
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:29.840528 18996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:29.840747 18996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:29.840768 18996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:29.841949 18996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:29.841995 18996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:29.842006 18996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:29.842020 18996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:29.842031 18996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:29.843400 18996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:29.843627 18996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:29.843973 18996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:29.848727 19001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:29.848711 19002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:29.848749 18996 server_base.cc:1034] running on GCE node
W20260926 13:18:29.848711 19004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:29.849251 18996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:29.849598 18996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:29.850762 18996 hybrid_clock.cc:648] HybridClock initialized: now 1790428709850745 us; error 41 us; skew 500 ppm
I20260926 13:18:29.852129 18996 webserver.cc:460] Webserver started at http://127.0.98.62:38577/ using document root <none> and password file <none>
I20260926 13:18:29.852370 18996 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:29.852448 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:29.852626 18996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:29.853708 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "123f784eeca54fec85b24dd56212ecfc"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "e9fc232314516283220c0bf1a37197a3"
server_key_iv: "5beb81dbe06778f3f80db040673bd8d8"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.854086 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "123f784eeca54fec85b24dd56212ecfc"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "e9fc232314516283220c0bf1a37197a3"
server_key_iv: "5beb81dbe06778f3f80db040673bd8d8"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.855383 18996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:29.856069 19010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:29.856237 18996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:29.856329 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "123f784eeca54fec85b24dd56212ecfc"
format_stamp: "Formatted at 2026-09-26 13:18:29 on dist-test-slave-f864"
server_key: "e9fc232314516283220c0bf1a37197a3"
server_key_iv: "5beb81dbe06778f3f80db040673bd8d8"
server_key_version: "encryptionkey@0"
I20260926 13:18:29.856407 18996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:29.868314 18996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:29.868602 18996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:29.868755 18996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:29.872781 18996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45979
I20260926 13:18:29.872815 19062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45979 every 8 connection(s)
I20260926 13:18:29.873152 18996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:29.874065 19063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:29.874413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 18996
I20260926 13:18:29.874516 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:29.874701 392 external_mini_cluster.cc:1428] Setting key c3d609093e7b48a9082621db895bbd89
I20260926 13:18:29.876840 19063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc: Bootstrap starting.
I20260926 13:18:29.877473 19063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:29.877766 19063 log.cc:824] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:29.878438 19063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc: No bootstrap required, opened a new log
I20260926 13:18:29.880123 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:29.875219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.0.98.62:45979)
0926 13:18:29.875415 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:29.875422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:29.875445 (+ 23us) server_negotiation.cc:407] Connection header received
0926 13:18:29.875850 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:29.875858 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:29.875918 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:29.876038 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:29.876270 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.877179 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.878248 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:29.878669 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:29.878754 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:29.879459 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:29.879463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:29.879463 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:29.879623 (+ 160us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:29.879653 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:29.879663 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:29.879719 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:29.879817 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:18:29.879949 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260926 13:18:29.880733 19063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123f784eeca54fec85b24dd56212ecfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260926 13:18:29.880890 19063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:29.880935 19063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f784eeca54fec85b24dd56212ecfc, State: Initialized, Role: FOLLOWER
I20260926 13:18:29.881054 19063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123f784eeca54fec85b24dd56212ecfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260926 13:18:29.881141 19063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:29.881183 19063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:29.881232 19063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:29.881757 19063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123f784eeca54fec85b24dd56212ecfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260926 13:18:29.881870 19063 leader_election.cc:302] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123f784eeca54fec85b24dd56212ecfc; no voters:
I20260926 13:18:29.882042 19063 leader_election.cc:288] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:29.882108 19068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:29.882306 19068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [term 1 LEADER]: Becoming Leader. State: Replica: 123f784eeca54fec85b24dd56212ecfc, State: Running, Role: LEADER
I20260926 13:18:29.882367 19063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:29.882429 19068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123f784eeca54fec85b24dd56212ecfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } }
I20260926 13:18:29.882778 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123f784eeca54fec85b24dd56212ecfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123f784eeca54fec85b24dd56212ecfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } } }
I20260926 13:18:29.882927 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:29.882798 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123f784eeca54fec85b24dd56212ecfc. Latest consensus state: current_term: 1 leader_uuid: "123f784eeca54fec85b24dd56212ecfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123f784eeca54fec85b24dd56212ecfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45979 } } }
I20260926 13:18:29.883359 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:29.883977 19077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:29.884514 19077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:29.886000 19077 catalog_manager.cc:1347] Generated new cluster ID: 9c124805abf4428dba757e235fa6775e
I20260926 13:18:29.886065 19077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:29.895449 19077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:29.895969 19077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:29.906584 19077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc: Generated new TSK 0
I20260926 13:18:29.906831 19077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:29.912632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--builtin_ntp_servers=127.0.98.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:29.994060 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:29.994326 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:29.994354 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:29.995512 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:29.995551 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:29.995576 19087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:29.996922 19087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:29.997164 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:29.997480 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:30.003491 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:30.003525 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:30.003471 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:30.003938 19087 server_base.cc:1034] running on GCE node
I20260926 13:18:30.004163 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:30.004398 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:30.005538 19087 hybrid_clock.cc:648] HybridClock initialized: now 1790428710005521 us; error 37 us; skew 500 ppm
I20260926 13:18:30.006837 19087 webserver.cc:460] Webserver started at http://127.0.98.1:32817/ using document root <none> and password file <none>
I20260926 13:18:30.007102 19087 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:30.007187 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:30.007331 19087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:30.008455 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "225c62d870fa4bdea30e605a58d3e4c3"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "be0f6ad968a479cd1593740696a55a0b"
server_key_iv: "f9e6b9479d6fbc425f9a2a40a60e2c9b"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.008802 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "225c62d870fa4bdea30e605a58d3e4c3"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "be0f6ad968a479cd1593740696a55a0b"
server_key_iv: "f9e6b9479d6fbc425f9a2a40a60e2c9b"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.010200 19087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:18:30.011168 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.011363 19087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:30.011461 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "225c62d870fa4bdea30e605a58d3e4c3"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "be0f6ad968a479cd1593740696a55a0b"
server_key_iv: "f9e6b9479d6fbc425f9a2a40a60e2c9b"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.011533 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:30.021054 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:30.021286 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:30.021382 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:30.021546 19087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:30.021858 19087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:30.021898 19087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.021952 19087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:30.021993 19087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.027520 19087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36047
I20260926 13:18:30.027563 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36047 every 8 connection(s)
I20260926 13:18:30.027899 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:30.032909 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.028614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47743 (local address 127.0.98.62:45979)
0926 13:18:30.028663 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.028666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.028913 (+ 247us) server_negotiation.cc:407] Connection header received
0926 13:18:30.029380 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.029381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.029390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.029430 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:30.030054 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.030602 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.031578 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.031758 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.031807 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.032685 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:30.032686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:30.032687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:30.032690 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:30.032704 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:30.032708 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:30.032749 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.032826 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.032850 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:18:30.033178 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.028614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45979 (local address 127.0.98.1:47743)
0926 13:18:30.028796 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.028829 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.029298 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.029490 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.029493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.029501 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:30.029989 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.029993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.030686 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.030687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.031497 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.031499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.031828 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.031829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.031946 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.032552 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:30.032553 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:30.032581 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:30.032604 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:30.032762 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:30.032766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:30.032768 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.032821 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.032908 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260926 13:18:30.033679 19215 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45979
I20260926 13:18:30.033780 19215 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:30.033986 19215 heartbeater.cc:511] Master 127.0.98.62:45979 requested a full tablet report, sending...
I20260926 13:18:30.034528 19027 ts_manager.cc:194] Registered new tserver with Master: 225c62d870fa4bdea30e605a58d3e4c3 (127.0.98.1:36047)
I20260926 13:18:30.035111 19027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47743
I20260926 13:18:30.037720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19087
I20260926 13:18:30.037807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:18:30.037979 392 external_mini_cluster.cc:1428] Setting key 942540f3428e53e73fb95e2cbc8f7021
I20260926 13:18:30.038884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--builtin_ntp_servers=127.0.98.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:30.117929 19218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:30.118160 19218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:30.118216 19218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:30.119455 19218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:30.119520 19218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:30.119562 19218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:30.120836 19218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:30.121104 19218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:30.121381 19218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:30.126551 19223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:30.126543 19226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:30.126577 19218 server_base.cc:1034] running on GCE node
W20260926 13:18:30.126523 19224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:30.126881 19218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:30.127125 19218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:30.128262 19218 hybrid_clock.cc:648] HybridClock initialized: now 1790428710128255 us; error 36 us; skew 500 ppm
I20260926 13:18:30.129519 19218 webserver.cc:460] Webserver started at http://127.0.98.2:39653/ using document root <none> and password file <none>
I20260926 13:18:30.129745 19218 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:30.129792 19218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:30.129915 19218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:30.130992 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "fcc407ed790e4ddeb7247e39976a0292"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "4afaf524d6c17c0af9988bcbde473305"
server_key_iv: "edd9e46aad156d54aa5c4108350123ef"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.131336 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fcc407ed790e4ddeb7247e39976a0292"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "4afaf524d6c17c0af9988bcbde473305"
server_key_iv: "edd9e46aad156d54aa5c4108350123ef"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.132612 19218 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:18:30.133291 19232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.133458 19218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:30.133548 19218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "fcc407ed790e4ddeb7247e39976a0292"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "4afaf524d6c17c0af9988bcbde473305"
server_key_iv: "edd9e46aad156d54aa5c4108350123ef"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.133628 19218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:30.147967 19218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:30.148269 19218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:30.148411 19218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:30.148638 19218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:30.148969 19218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:30.149024 19218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.149065 19218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:30.149093 19218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.154925 19218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39997
I20260926 13:18:30.154971 19345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39997 every 8 connection(s)
I20260926 13:18:30.155294 19218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:18:30.160455 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.155994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54141 (local address 127.0.98.62:45979)
0926 13:18:30.156041 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.156044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.156289 (+ 245us) server_negotiation.cc:407] Connection header received
0926 13:18:30.156776 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.156777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.156787 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.156825 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:30.157483 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.157969 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.159191 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.159372 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.159425 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.160216 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:30.160217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:30.160218 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:30.160224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:30.160240 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:30.160245 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:30.160297 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.160375 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.160396 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:18:30.160698 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.155992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45979 (local address 127.0.98.2:54141)
0926 13:18:30.156173 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.156205 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.156683 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.156895 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.156898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.156906 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:30.157416 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.157420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.158068 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.158070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.159018 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.159020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.159443 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.159444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.159563 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.160074 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:30.160074 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:30.160104 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:30.160126 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:30.160306 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:30.160310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:30.160312 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.160374 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.160458 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260926 13:18:30.161296 19346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45979
I20260926 13:18:30.161432 19346 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:30.161617 19346 heartbeater.cc:511] Master 127.0.98.62:45979 requested a full tablet report, sending...
I20260926 13:18:30.162017 19027 ts_manager.cc:194] Registered new tserver with Master: fcc407ed790e4ddeb7247e39976a0292 (127.0.98.2:39997)
I20260926 13:18:30.162420 19027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54141
I20260926 13:18:30.163283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19218
I20260926 13:18:30.163388 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:18:30.163537 392 external_mini_cluster.cc:1428] Setting key 60d0df0efceb5620d3b2a1e1f46d192f
I20260926 13:18:30.164335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--builtin_ntp_servers=127.0.98.20:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:30.242681 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:30.242897 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:30.242933 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:30.244055 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:30.244112 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:30.244153 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:30.245536 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:30.245788 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:30.246060 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:30.250676 19357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:30.250706 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:30.250972 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:30.251072 19349 server_base.cc:1034] running on GCE node
I20260926 13:18:30.251279 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:30.251535 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:30.252699 19349 hybrid_clock.cc:648] HybridClock initialized: now 1790428710252686 us; error 24 us; skew 500 ppm
I20260926 13:18:30.253873 19349 webserver.cc:460] Webserver started at http://127.0.98.3:35119/ using document root <none> and password file <none>
I20260926 13:18:30.254103 19349 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:30.254186 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:30.254323 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:30.255357 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9da568a7fcd4f0d900e343e59bc5e73"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "601bdaff7882387349da79d8d5a7bb1f"
server_key_iv: "754bb1f49d95f4b865ab00107d880e98"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.255712 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9da568a7fcd4f0d900e343e59bc5e73"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "601bdaff7882387349da79d8d5a7bb1f"
server_key_iv: "754bb1f49d95f4b865ab00107d880e98"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.256945 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:30.257612 19363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.257771 19349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:30.257862 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "f9da568a7fcd4f0d900e343e59bc5e73"
format_stamp: "Formatted at 2026-09-26 13:18:30 on dist-test-slave-f864"
server_key: "601bdaff7882387349da79d8d5a7bb1f"
server_key_iv: "754bb1f49d95f4b865ab00107d880e98"
server_key_version: "encryptionkey@0"
I20260926 13:18:30.257941 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:30.266224 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:30.266479 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:30.266615 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:30.266824 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:30.267125 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:30.267174 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.267210 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:30.267236 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:30.272753 19349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45317
I20260926 13:18:30.272820 19476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45317 every 8 connection(s)
I20260926 13:18:30.273136 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:18:30.278226 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.273842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38759 (local address 127.0.98.62:45979)
0926 13:18:30.273884 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.273886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.274139 (+ 253us) server_negotiation.cc:407] Connection header received
0926 13:18:30.274691 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.274693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.274704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.274767 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:30.275428 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.275919 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.276868 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.277096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.277177 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.277929 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:30.277930 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:30.277931 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:30.277936 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:30.277949 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:30.277953 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:30.278001 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.278097 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.278122 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:18:30.278438 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.273852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45979 (local address 127.0.98.3:38759)
0926 13:18:30.274030 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.274061 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.274589 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.274835 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.274840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.274854 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:30.275331 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.275335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.276021 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.276022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.276783 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.276785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.277186 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.277187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.277313 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.277801 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:30.277801 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:30.277827 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:30.277847 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:30.278017 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:30.278026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:30.278029 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.278098 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.278220 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260926 13:18:30.278613 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19349
I20260926 13:18:30.278709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:18:30.278893 392 external_mini_cluster.cc:1428] Setting key 4a31f0d552a8125963f053f2ff8d9135
I20260926 13:18:30.278983 19477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45979
I20260926 13:18:30.279095 19477 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:30.279278 19477 heartbeater.cc:511] Master 127.0.98.62:45979 requested a full tablet report, sending...
I20260926 13:18:30.279780 19027 ts_manager.cc:194] Registered new tserver with Master: f9da568a7fcd4f0d900e343e59bc5e73 (127.0.98.3:45317)
I20260926 13:18:30.280143 19027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38759
I20260926 13:18:30.290809 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:18:30.295208 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.292372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.0.98.62:45979)
0926 13:18:30.292428 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.292430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.292526 (+ 96us) server_negotiation.cc:407] Connection header received
0926 13:18:30.292587 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.292587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.292596 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.292633 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:30.292753 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.293208 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.293945 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.294140 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.294280 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.294901 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:30.294902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:30.294903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:30.294906 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:30.294919 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:30.294923 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:30.295005 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.295069 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.295079 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:18:30.296634 19027 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:18:30.298174 19027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:18:30.304579 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.300342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.3:45317)
0926 13:18:30.300790 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.300794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.300803 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:18:30.300837 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.300839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.300852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.300940 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:30.301144 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.302305 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.303476 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.304174 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.304277 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.304376 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.304448 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.304490 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":80,"threads_started":1}
I20260926 13:18:30.304581 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.300258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45317 (local address 127.0.0.1:53982)
0926 13:18:30.300608 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.300619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.300671 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.301021 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.301022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.301024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:30.301063 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.301065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.302394 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.302395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.303390 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.303392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.304276 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.304277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.304349 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.304364 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.304435 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.304490 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":27,"thread_start_us":100,"threads_started":1}
I20260926 13:18:30.304711 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.300655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46736 (local address 127.0.98.1:36047)
0926 13:18:30.301019 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.301023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.301033 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:18:30.301071 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.301073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.301088 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.301158 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:30.301321 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.301980 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.303687 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.304401 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.304471 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.304546 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.304580 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.304620 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":91,"threads_started":1}
I20260926 13:18:30.304693 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.300446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36047 (local address 127.0.0.1:46736)
0926 13:18:30.300836 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.300846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.300890 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.301225 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.301226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.301228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:30.301264 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.301265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.302157 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.302192 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.303599 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.303602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.304472 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.304473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.304521 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.304524 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.304560 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.304591 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":62,"thread_start_us":79,"threads_started":1}
I20260926 13:18:30.305264 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.300441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.0.98.2:39997)
0926 13:18:30.300593 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.300597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.300609 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:18:30.300653 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.300655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.300673 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.300761 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:30.300961 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.302260 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.303705 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.304180 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.304272 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.304376 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.305141 (+ 765us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.305182 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260926 13:18:30.305224 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.300256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39997 (local address 127.0.0.1:47970)
0926 13:18:30.300414 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.300454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.300528 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.300774 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.300777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.300780 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:30.300847 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.300849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.302375 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.302376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.303603 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.303605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.304998 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.304999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.305043 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.305046 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.305121 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.305161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":70,"thread_start_us":52,"threads_started":1}
I20260926 13:18:30.306036 19411 tablet_service.cc:1467] Processing CreateTablet for tablet 289d4cfd9e97491c822ce52be8eaf9f5 (DEFAULT_TABLE table=test-table [id=41bfc8d6403b489bad347ae5a1fcb1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:30.306372 19149 tablet_service.cc:1467] Processing CreateTablet for tablet 289d4cfd9e97491c822ce52be8eaf9f5 (DEFAULT_TABLE table=test-table [id=41bfc8d6403b489bad347ae5a1fcb1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:30.306491 19411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d4cfd9e97491c822ce52be8eaf9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:30.306401 19280 tablet_service.cc:1467] Processing CreateTablet for tablet 289d4cfd9e97491c822ce52be8eaf9f5 (DEFAULT_TABLE table=test-table [id=41bfc8d6403b489bad347ae5a1fcb1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:30.306708 19149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d4cfd9e97491c822ce52be8eaf9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:30.306731 19280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289d4cfd9e97491c822ce52be8eaf9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:30.309113 19491 tablet_bootstrap.cc:492] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: Bootstrap starting.
I20260926 13:18:30.309777 19491 tablet_bootstrap.cc:654] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:30.309799 19493 tablet_bootstrap.cc:492] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Bootstrap starting.
I20260926 13:18:30.310248 19491 log.cc:824] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:30.310352 19492 tablet_bootstrap.cc:492] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Bootstrap starting.
I20260926 13:18:30.310473 19493 tablet_bootstrap.cc:654] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:30.310799 19493 log.cc:824] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:30.310990 19492 tablet_bootstrap.cc:654] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:30.311056 19491 tablet_bootstrap.cc:492] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: No bootstrap required, opened a new log
I20260926 13:18:30.311131 19491 ts_tablet_manager.cc:1397] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:30.311314 19492 log.cc:824] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:30.311554 19493 tablet_bootstrap.cc:492] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: No bootstrap required, opened a new log
I20260926 13:18:30.311628 19493 ts_tablet_manager.cc:1397] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:18:30.312083 19492 tablet_bootstrap.cc:492] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: No bootstrap required, opened a new log
I20260926 13:18:30.312191 19492 ts_tablet_manager.cc:1397] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:30.313421 19493 raft_consensus.cc:348] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.313546 19493 raft_consensus.cc:374] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:30.313567 19493 raft_consensus.cc:729] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc407ed790e4ddeb7247e39976a0292, State: Initialized, Role: FOLLOWER
I20260926 13:18:30.313705 19493 consensus_queue.cc:260] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.313971 19493 ts_tablet_manager.cc:1428] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 13:18:30.313886 19491 raft_consensus.cc:348] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.314040 19346 heartbeater.cc:503] Master 127.0.98.62:45979 was elected leader, sending a full tablet report...
I20260926 13:18:30.314036 19491 raft_consensus.cc:374] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:30.314080 19491 raft_consensus.cc:729] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225c62d870fa4bdea30e605a58d3e4c3, State: Initialized, Role: FOLLOWER
I20260926 13:18:30.314234 19491 consensus_queue.cc:260] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.314507 19491 ts_tablet_manager.cc:1428] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 13:18:30.314631 19215 heartbeater.cc:503] Master 127.0.98.62:45979 was elected leader, sending a full tablet report...
I20260926 13:18:30.315590 19492 raft_consensus.cc:348] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.315709 19492 raft_consensus.cc:374] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:30.315729 19492 raft_consensus.cc:729] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9da568a7fcd4f0d900e343e59bc5e73, State: Initialized, Role: FOLLOWER
I20260926 13:18:30.315867 19492 consensus_queue.cc:260] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.316128 19492 ts_tablet_manager.cc:1428] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 13:18:30.316176 19477 heartbeater.cc:503] Master 127.0.98.62:45979 was elected leader, sending a full tablet report...
W20260926 13:18:30.405848 19347 tablet.cc:2369] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:18:30.523684 19478 tablet.cc:2369] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:18:30.528553 19216 tablet.cc:2369] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:18:30.619747 19498 raft_consensus.cc:482] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:30.619962 19498 raft_consensus.cc:504] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.620380 19498 leader_election.cc:288] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcc407ed790e4ddeb7247e39976a0292 (127.0.98.2:39997), f9da568a7fcd4f0d900e343e59bc5e73 (127.0.98.3:45317)
I20260926 13:18:30.623338 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.620655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55659 (local address 127.0.98.2:39997)
0926 13:18:30.620731 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.620733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.620816 (+ 83us) server_negotiation.cc:407] Connection header received
0926 13:18:30.620898 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.620898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.620908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.620985 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:30.621168 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.621645 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.622629 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.623009 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.623064 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.623072 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.623222 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.623272 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:18:30.623342 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.620585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39997 (local address 127.0.98.1:55659)
0926 13:18:30.620722 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.620734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.620807 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.621005 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.621006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.621008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:30.621081 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.621083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.621749 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.621749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.622546 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.622547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.623083 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.623084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.623153 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.623157 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.623206 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.623234 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":2,"thread_start_us":74,"threads_started":1}
I20260926 13:18:30.623461 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.620593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45317 (local address 127.0.98.1:49617)
0926 13:18:30.620733 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.620742 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:30.620807 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:30.620972 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:30.620974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:30.620976 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:30.621051 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.621053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.621710 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.621711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.622605 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:30.622607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.623138 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.623139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:30.623206 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.623211 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 13:18:30.623267 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 13:18:30.623300 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":22,"thread_start_us":64,"threads_started":1}
I20260926 13:18:30.623410 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.620679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49617 (local address 127.0.98.3:45317)
0926 13:18:30.620768 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.620770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.620781 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:18:30.620887 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.620888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.620898 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.620951 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:30.621124 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.621629 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.622691 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.623032 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.623119 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.623128 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.623285 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.623324 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 13:18:30.623768 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d4cfd9e97491c822ce52be8eaf9f5" candidate_uuid: "225c62d870fa4bdea30e605a58d3e4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc407ed790e4ddeb7247e39976a0292" is_pre_election: true
I20260926 13:18:30.623925 19300 raft_consensus.cc:2454] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225c62d870fa4bdea30e605a58d3e4c3 in term 0.
I20260926 13:18:30.624085 19431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d4cfd9e97491c822ce52be8eaf9f5" candidate_uuid: "225c62d870fa4bdea30e605a58d3e4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" is_pre_election: true
I20260926 13:18:30.624157 19104 leader_election.cc:302] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225c62d870fa4bdea30e605a58d3e4c3, fcc407ed790e4ddeb7247e39976a0292; no voters:
I20260926 13:18:30.624181 19431 raft_consensus.cc:2454] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 225c62d870fa4bdea30e605a58d3e4c3 in term 0.
I20260926 13:18:30.624262 19498 raft_consensus.cc:2781] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:18:30.624317 19498 raft_consensus.cc:482] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:18:30.624342 19498 raft_consensus.cc:3037] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:30.625032 19498 raft_consensus.cc:504] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.625166 19498 leader_election.cc:288] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [CANDIDATE]: Term 1 election: Requested vote from peers fcc407ed790e4ddeb7247e39976a0292 (127.0.98.2:39997), f9da568a7fcd4f0d900e343e59bc5e73 (127.0.98.3:45317)
I20260926 13:18:30.625308 19431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d4cfd9e97491c822ce52be8eaf9f5" candidate_uuid: "225c62d870fa4bdea30e605a58d3e4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9da568a7fcd4f0d900e343e59bc5e73"
I20260926 13:18:30.625375 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289d4cfd9e97491c822ce52be8eaf9f5" candidate_uuid: "225c62d870fa4bdea30e605a58d3e4c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcc407ed790e4ddeb7247e39976a0292"
I20260926 13:18:30.625393 19431 raft_consensus.cc:3037] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:30.625468 19300 raft_consensus.cc:3037] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:30.626209 19431 raft_consensus.cc:2454] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225c62d870fa4bdea30e605a58d3e4c3 in term 1.
I20260926 13:18:30.626348 19300 raft_consensus.cc:2454] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 225c62d870fa4bdea30e605a58d3e4c3 in term 1.
I20260926 13:18:30.626385 19105 leader_election.cc:302] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225c62d870fa4bdea30e605a58d3e4c3, f9da568a7fcd4f0d900e343e59bc5e73; no voters:
I20260926 13:18:30.626500 19498 raft_consensus.cc:2781] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:30.626633 19498 raft_consensus.cc:686] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 225c62d870fa4bdea30e605a58d3e4c3, State: Running, Role: LEADER
I20260926 13:18:30.626736 19498 consensus_queue.cc:237] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } }
I20260926 13:18:30.627562 19027 catalog_manager.cc:5495] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 225c62d870fa4bdea30e605a58d3e4c3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "225c62d870fa4bdea30e605a58d3e4c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "225c62d870fa4bdea30e605a58d3e4c3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36047 } health_report { overall_health: HEALTHY } } }
I20260926 13:18:30.657770 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.655644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.0.98.1:36047)
0926 13:18:30.655705 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:30.655707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:30.655715 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:18:30.655751 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:30.655752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:30.655772 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:30.655832 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:18:30.655953 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.656411 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.657121 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:30.657293 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:30.657340 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:30.657460 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:18:30.657542 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:18:30.657611 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:30.657653 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 13:18:30.657662 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:18:30.659713 19431 raft_consensus.cc:1261] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 1 FOLLOWER]: Refusing update from remote peer 225c62d870fa4bdea30e605a58d3e4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:30.659713 19300 raft_consensus.cc:1261] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 1 FOLLOWER]: Refusing update from remote peer 225c62d870fa4bdea30e605a58d3e4c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:30.660084 19502 consensus_queue.cc:1035] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcc407ed790e4ddeb7247e39976a0292" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:30.660226 19498 consensus_queue.cc:1035] T 289d4cfd9e97491c822ce52be8eaf9f5 P 225c62d870fa4bdea30e605a58d3e4c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9da568a7fcd4f0d900e343e59bc5e73" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:30.662451 19505 mvcc.cc:205] Tried to move back new op lower bound from 7333595998859202560 to 7333595998727823360. Current Snapshot: MvccSnapshot[applied={T|T < 7333595998859202560}]
I20260926 13:18:30.662999 19507 mvcc.cc:205] Tried to move back new op lower bound from 7333595998859202560 to 7333595998727823360. Current Snapshot: MvccSnapshot[applied={T|T < 7333595998859202560}]
I20260926 13:18:30.663280 19506 mvcc.cc:205] Tried to move back new op lower bound from 7333595998859202560 to 7333595998727823360. Current Snapshot: MvccSnapshot[applied={T|T < 7333595998859202560}]
I20260926 13:18:30.664857 19027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:30.664933 19027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:30.665653 19027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 123f784eeca54fec85b24dd56212ecfc: Sending DeleteTablet for 3 replicas of tablet 289d4cfd9e97491c822ce52be8eaf9f5
I20260926 13:18:30.665988 19280 tablet_service.cc:1514] Processing DeleteTablet for tablet 289d4cfd9e97491c822ce52be8eaf9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:30 UTC) from {username='slave'} at 127.0.0.1:47970
I20260926 13:18:30.666069 19411 tablet_service.cc:1514] Processing DeleteTablet for tablet 289d4cfd9e97491c822ce52be8eaf9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:30 UTC) from {username='slave'} at 127.0.0.1:53982
I20260926 13:18:30.666213 19149 tablet_service.cc:1514] Processing DeleteTablet for tablet 289d4cfd9e97491c822ce52be8eaf9f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:30 UTC) from {username='slave'} at 127.0.0.1:46736
I20260926 13:18:30.666250 19515 tablet_replica.cc:330] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: stopping tablet replica
I20260926 13:18:30.666391 19515 raft_consensus.cc:2229] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:30.666502 19515 raft_consensus.cc:2258] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:30.666621 19516 tablet_replica.cc:330] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: stopping tablet replica
I20260926 13:18:30.666719 19516 raft_consensus.cc:2229] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:30.666718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19087
I20260926 13:18:30.666854 19516 raft_consensus.cc:2258] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:30.668133 19516 ts_tablet_manager.cc:1905] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:30.669406 19515 ts_tablet_manager.cc:1905] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:30.670068 19516 ts_tablet_manager.cc:1918] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:30.670143 19516 log.cc:1196] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/289d4cfd9e97491c822ce52be8eaf9f5
I20260926 13:18:30.670428 19516 ts_tablet_manager.cc:1939] T 289d4cfd9e97491c822ce52be8eaf9f5 P f9da568a7fcd4f0d900e343e59bc5e73: Deleting consensus metadata
I20260926 13:18:30.670836 19014 catalog_manager.cc:4841] TS f9da568a7fcd4f0d900e343e59bc5e73 (127.0.98.3:45317): tablet 289d4cfd9e97491c822ce52be8eaf9f5 (table test-table [id=41bfc8d6403b489bad347ae5a1fcb1c3]) successfully deleted
I20260926 13:18:30.671056 19515 ts_tablet_manager.cc:1918] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:30.671110 19515 log.cc:1196] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790428444292953-392-0/minicluster-data/ts-1/wal/wals/289d4cfd9e97491c822ce52be8eaf9f5
I20260926 13:18:30.671334 19515 ts_tablet_manager.cc:1939] T 289d4cfd9e97491c822ce52be8eaf9f5 P fcc407ed790e4ddeb7247e39976a0292: Deleting consensus metadata
W20260926 13:18:30.671614 19013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36047 (error 108)
I20260926 13:18:30.671839 19013 catalog_manager.cc:4841] TS fcc407ed790e4ddeb7247e39976a0292 (127.0.98.2:39997): tablet 289d4cfd9e97491c822ce52be8eaf9f5 (table test-table [id=41bfc8d6403b489bad347ae5a1fcb1c3]) successfully deleted
I20260926 13:18:30.671969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19218
I20260926 13:18:30.672125 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:30.671992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36047 (local address 127.0.0.1:46754)
0926 13:18:30.672044 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:30.672087 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36047: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:18:30.672713 19013 catalog_manager.cc:4584] TS 225c62d870fa4bdea30e605a58d3e4c3 (127.0.98.1:36047): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 289d4cfd9e97491c822ce52be8eaf9f5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36047: connect: Connection refused (error 111)
I20260926 13:18:30.676793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19349
I20260926 13:18:30.681344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 18996
2026-09-26T13:18:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (947 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:30 dist-test-slave-f864 krb5kdc[19521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:30 dist-test-slave-f864 krb5kdc[19521](info): set up 2 sockets
Sep 26 13:18:30 dist-test-slave-f864 krb5kdc[19521](info): commencing operation
krb5kdc: starting...
W20260926 13:18:31.714025 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:31 dist-test-slave-f864 krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:31.829175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40261
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:31.909168 19534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:31.909435 19534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:31.909478 19534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:31.910681 19534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:31.910753 19534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:31.910775 19534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:31.910795 19534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:31.910849 19534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:31.912330 19534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:31.912596 19534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:31.912825 19534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:31.917692 19542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:31.917704 19540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:31.917744 19539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:31.917917 19534 server_base.cc:1034] running on GCE node
I20260926 13:18:31.918249 19534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:31.918561 19534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:31.919708 19534 hybrid_clock.cc:648] HybridClock initialized: now 1790428711919685 us; error 36 us; skew 500 ppm
Sep 26 13:18:31 dist-test-slave-f864 krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:31.922132 19534 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:31.922724 19534 webserver.cc:460] Webserver started at http://127.0.98.62:43297/ using document root <none> and password file <none>
I20260926 13:18:31.922947 19534 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:31.922992 19534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:31.923141 19534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:31.924175 19534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "8467a06a338f400b9ff35cc257c76d65"
format_stamp: "Formatted at 2026-09-26 13:18:31 on dist-test-slave-f864"
server_key: "230fe479eb5f672cb041bd4da8978329"
server_key_iv: "f493995947eb9b88666b4391f5cccf37"
server_key_version: "encryptionkey@0"
I20260926 13:18:31.924533 19534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "8467a06a338f400b9ff35cc257c76d65"
format_stamp: "Formatted at 2026-09-26 13:18:31 on dist-test-slave-f864"
server_key: "230fe479eb5f672cb041bd4da8978329"
server_key_iv: "f493995947eb9b88666b4391f5cccf37"
server_key_version: "encryptionkey@0"
I20260926 13:18:31.925824 19534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:18:31.926573 19549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:31.926724 19534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:31.926807 19534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "8467a06a338f400b9ff35cc257c76d65"
format_stamp: "Formatted at 2026-09-26 13:18:31 on dist-test-slave-f864"
server_key: "230fe479eb5f672cb041bd4da8978329"
server_key_iv: "f493995947eb9b88666b4391f5cccf37"
server_key_version: "encryptionkey@0"
I20260926 13:18:31.926859 19534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:31.935845 19534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:31.938323 19534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:31.938447 19534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:31.942535 19534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40261
I20260926 13:18:31.942564 19601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40261 every 8 connection(s)
I20260926 13:18:31.942950 19534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:31.943507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19534
I20260926 13:18:31.943631 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:31.943840 392 external_mini_cluster.cc:1428] Setting key 0925ce53c1754d069a6b976782bda903
I20260926 13:18:31.943972 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:31.947284 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: Bootstrap starting.
I20260926 13:18:31.947974 19602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:31.948283 19602 log.cc:824] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:31.948944 19602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: No bootstrap required, opened a new log
Sep 26 13:18:31 dist-test-slave-f864 krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:31.950970 19602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8467a06a338f400b9ff35cc257c76d65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260926 13:18:31.951130 19602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:31.951177 19602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8467a06a338f400b9ff35cc257c76d65, State: Initialized, Role: FOLLOWER
I20260926 13:18:31.951344 19602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8467a06a338f400b9ff35cc257c76d65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260926 13:18:31.951439 19602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:31.951472 19602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:31.951527 19602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:31.952170 19602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8467a06a338f400b9ff35cc257c76d65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260926 13:18:31.952261 19602 leader_election.cc:302] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8467a06a338f400b9ff35cc257c76d65; no voters:
I20260926 13:18:31.952442 19602 leader_election.cc:288] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:31.952472 19607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:31.952592 19607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [term 1 LEADER]: Becoming Leader. State: Replica: 8467a06a338f400b9ff35cc257c76d65, State: Running, Role: LEADER
I20260926 13:18:31.952682 19607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8467a06a338f400b9ff35cc257c76d65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } }
I20260926 13:18:31.952723 19602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:31.953017 19607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8467a06a338f400b9ff35cc257c76d65. Latest consensus state: current_term: 1 leader_uuid: "8467a06a338f400b9ff35cc257c76d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8467a06a338f400b9ff35cc257c76d65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } } }
I20260926 13:18:31.953027 19608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8467a06a338f400b9ff35cc257c76d65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8467a06a338f400b9ff35cc257c76d65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40261 } } }
I20260926 13:18:31.953099 19607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:31.953127 19608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65 [sys.catalog]: This master's current role is: LEADER
W20260926 13:18:31.954597 19623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:18:31.954649 19623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:18:31.954694 19613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:31.954838 19613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:31.954928 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:31.944389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60150 (local address 127.0.98.62:40261)
0926 13:18:31.944603 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:31.944611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:31.944636 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:18:31.945095 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:31.945102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:31.945151 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:31.945268 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:31.946059 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:31.947087 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:31.948180 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:31.948624 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:31.948777 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:31.951070 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:31.951074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:31.951074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:31.951084 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:31.953156 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:31.953585 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:31.953588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:31.953588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:31.953686 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:31.953912 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:31.953913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:31.953913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:31.954269 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:31.954439 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:31.954564 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 13:18:31.954697 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":101,"threads_started":1}
I20260926 13:18:31.956459 19613 catalog_manager.cc:1347] Generated new cluster ID: 1365276bebb14825a1e79f7c43f9f07d
I20260926 13:18:31.956501 19613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:31.960239 19613 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:31.960700 19613 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:31.964325 19613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: Generated new TSK 0
I20260926 13:18:31.964497 19613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 13:18:31.996337 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:32.077981 19628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:32.078228 19628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:32.078281 19628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:32.079367 19628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:32.079403 19628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:32.079428 19628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:32.080848 19628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:32.081066 19628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:32.081286 19628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:18:32.086297 19628 server_base.cc:1034] running on GCE node
W20260926 13:18:32.086298 19633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.086402 19634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.086282 19636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:32.086787 19628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:32.087046 19628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:32.088199 19628 hybrid_clock.cc:648] HybridClock initialized: now 1790428712088186 us; error 32 us; skew 500 ppm
Sep 26 13:18:32 dist-test-slave-f864 krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:32.090754 19628 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 13:18:32.091172 19628 webserver.cc:460] Webserver started at http://127.0.98.1:40361/ using document root <none> and password file <none>
I20260926 13:18:32.091391 19628 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:32.091467 19628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:32.091624 19628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:32.092628 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "566949a4e15a4504aa4c6767184c8732"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "804c26aa560dd2202273bfa59182d89e"
server_key_iv: "bc65b22a62d7eddb452a927b483fc974"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.092972 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "566949a4e15a4504aa4c6767184c8732"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "804c26aa560dd2202273bfa59182d89e"
server_key_iv: "bc65b22a62d7eddb452a927b483fc974"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.094292 19628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:32.095023 19643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.095250 19628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:32.095353 19628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "566949a4e15a4504aa4c6767184c8732"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "804c26aa560dd2202273bfa59182d89e"
server_key_iv: "bc65b22a62d7eddb452a927b483fc974"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.095436 19628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:32.111347 19628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:32.113216 19628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:32.113375 19628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:32.113606 19628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:32.113905 19628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:32.113963 19628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.114002 19628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:32.114030 19628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.119722 19628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32963
I20260926 13:18:32.119762 19756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32963 every 8 connection(s)
I20260926 13:18:32.120085 19628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:32.122087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19628
I20260926 13:18:32.122224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:18:32.122439 392 external_mini_cluster.cc:1428] Setting key aa660c807c27f80a0859958fbba8f2b4
Sep 26 13:18:32 dist-test-slave-f864 krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 13:18:32.130621 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.120883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44357 (local address 127.0.98.62:40261)
0926 13:18:32.120970 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.120972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.121128 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:18:32.122218 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.122220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.122233 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.122304 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.123150 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.123629 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.125096 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.125359 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.125390 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.127657 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.127658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.127659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:32.127666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.129226 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.129622 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.129623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.129624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.129680 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.129908 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.129909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.129909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.130027 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.130104 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.130434 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.130529 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 13:18:32.130771 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.120796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40261 (local address 127.0.98.1:44357)
0926 13:18:32.120987 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.121028 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.122094 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.122397 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.122401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.122742 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:32.123074 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.123081 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.123723 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.123724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.124989 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.124992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.125464 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.125465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.125586 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.126091 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:32.126107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:32.127567 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:32.129307 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:32.129314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:32.129332 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:32.129550 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:32.129731 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:32.129732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:32.129732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:32.129838 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:32.130103 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:32.130111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:32.130292 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.130406 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.130556 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260926 13:18:32.131361 19757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40261
I20260926 13:18:32.131474 19757 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:32.131640 19757 heartbeater.cc:511] Master 127.0.98.62:40261 requested a full tablet report, sending...
I20260926 13:18:32.132143 19566 ts_manager.cc:194] Registered new tserver with Master: 566949a4e15a4504aa4c6767184c8732 (127.0.98.1:32963)
I20260926 13:18:32.132730 19566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 13:18:32.149912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:32.230491 19762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:32.230700 19762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:32.230724 19762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:32.232399 19762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:32.232453 19762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:32.232498 19762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:32.233986 19762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:32.234284 19762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:32.234516 19762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:32.239401 19770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.239401 19767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.239408 19768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:32.239686 19762 server_base.cc:1034] running on GCE node
I20260926 13:18:32.239885 19762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:32.240115 19762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:32.241258 19762 hybrid_clock.cc:648] HybridClock initialized: now 1790428712241247 us; error 28 us; skew 500 ppm
Sep 26 13:18:32 dist-test-slave-f864 krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:32.243690 19762 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 13:18:32.244055 19762 webserver.cc:460] Webserver started at http://127.0.98.2:42161/ using document root <none> and password file <none>
I20260926 13:18:32.244231 19762 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:32.244269 19762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:32.244352 19762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:32.245339 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "48b4d23c985f4f9a977335d4b8056726"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "e50ed558ace0c94712c36bfe7a825d3e"
server_key_iv: "135402b68f6733dbaabb09661ab60d5e"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.245666 19762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48b4d23c985f4f9a977335d4b8056726"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "e50ed558ace0c94712c36bfe7a825d3e"
server_key_iv: "135402b68f6733dbaabb09661ab60d5e"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.246948 19762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:32.247653 19777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.247874 19762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:32.247969 19762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "48b4d23c985f4f9a977335d4b8056726"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "e50ed558ace0c94712c36bfe7a825d3e"
server_key_iv: "135402b68f6733dbaabb09661ab60d5e"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.248047 19762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:32.267692 19762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:32.269999 19762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:32.270193 19762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:32.270447 19762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:32.270766 19762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:32.270817 19762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.270856 19762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:32.270885 19762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.276557 19762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40141
I20260926 13:18:32.276657 19890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40141 every 8 connection(s)
I20260926 13:18:32.276954 19762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 13:18:32 dist-test-slave-f864 krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:32.285169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19762
I20260926 13:18:32.285301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:18:32.285544 392 external_mini_cluster.cc:1428] Setting key cf24ff7286cae36d38e941d450a87714
I20260926 13:18:32.287061 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.277684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60933 (local address 127.0.98.62:40261)
0926 13:18:32.277729 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.277732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.277927 (+ 195us) server_negotiation.cc:407] Connection header received
0926 13:18:32.278922 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.278923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.278940 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.278997 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.279635 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.280199 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.281180 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.281409 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.281476 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.283869 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.283871 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.283871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:32.283880 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.285551 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.285971 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.285973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.285973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.286040 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.286372 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.286373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.286373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.286500 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.286581 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.286855 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.286973 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:18:32.287220 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.277632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40261 (local address 127.0.98.2:60933)
0926 13:18:32.277789 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.277831 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.278822 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.279079 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.279083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.279330 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:32.279515 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.279519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.280287 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.280287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.281086 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.281088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.281491 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.281492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.281593 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.282222 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:32.282234 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:32.283764 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:32.285663 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:32.285670 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:32.285671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:32.285883 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:32.286109 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:32.286110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:32.286110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:32.286284 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:32.286574 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:32.286581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:32.286719 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.286822 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.286985 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260926 13:18:32.287727 19891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40261
I20260926 13:18:32.287834 19891 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:32.288040 19891 heartbeater.cc:511] Master 127.0.98.62:40261 requested a full tablet report, sending...
I20260926 13:18:32.288415 19566 ts_manager.cc:194] Registered new tserver with Master: 48b4d23c985f4f9a977335d4b8056726 (127.0.98.2:40141)
I20260926 13:18:32.288836 19566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60933
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 13:18:32.311519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:32.396987 19896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:32.397184 19896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:32.397244 19896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:32.398914 19896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:32.398975 19896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:32.399024 19896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:32.400894 19896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:32.401158 19896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:32.401389 19896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:32.406399 19902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.406404 19904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.406406 19901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:32.406628 19896 server_base.cc:1034] running on GCE node
I20260926 13:18:32.406841 19896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:32.407080 19896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:32.408237 19896 hybrid_clock.cc:648] HybridClock initialized: now 1790428712408221 us; error 50 us; skew 500 ppm
Sep 26 13:18:32 dist-test-slave-f864 krb5kdc[19521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:32.410856 19896 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 13:18:32.411278 19896 webserver.cc:460] Webserver started at http://127.0.98.3:36757/ using document root <none> and password file <none>
I20260926 13:18:32.411482 19896 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:32.411523 19896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:32.411612 19896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:32.412611 19896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "3a68154f26cc43059f42e3ff5a716091"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "5c4eb877abb12aa958f764ae18c804ab"
server_key_iv: "72d969bc142d26fcf0bef6e7565aa5dd"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.412902 19896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3a68154f26cc43059f42e3ff5a716091"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "5c4eb877abb12aa958f764ae18c804ab"
server_key_iv: "72d969bc142d26fcf0bef6e7565aa5dd"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.414132 19896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:32.414880 19911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.415040 19896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:32.415126 19896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "3a68154f26cc43059f42e3ff5a716091"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "5c4eb877abb12aa958f764ae18c804ab"
server_key_iv: "72d969bc142d26fcf0bef6e7565aa5dd"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.415195 19896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:32.423036 19896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:32.424880 19896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:32.425034 19896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:32.425240 19896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:32.425560 19896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:32.425616 19896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.425654 19896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:32.425683 19896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.431448 19896 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44969
I20260926 13:18:32.431519 20024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44969 every 8 connection(s)
I20260926 13:18:32.431835 19896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:18:32.436547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 19896
I20260926 13:18:32.436661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:18:32.436841 392 external_mini_cluster.cc:1428] Setting key 7664925d819b008372dd4e8432e22e81
Sep 26 13:18:32 dist-test-slave-f864 krb5kdc[19521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:32.440824 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.432581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46083 (local address 127.0.98.62:40261)
0926 13:18:32.432637 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.432639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.432814 (+ 175us) server_negotiation.cc:407] Connection header received
0926 13:18:32.433756 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.433757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.433766 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.433819 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.434474 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.434986 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.436000 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.436182 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.436231 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.438543 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.438545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.438545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:32.438551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.439770 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.440115 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.440116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.440116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.440164 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.440357 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.440357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.440357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.440452 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.440502 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.440708 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.440764 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 13:18:32.441021 20027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.432564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40261 (local address 127.0.98.3:46083)
0926 13:18:32.432716 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.432749 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.433663 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.433871 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.433875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.434142 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:32.434401 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.434406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.435059 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.435060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.435914 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.435917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.436245 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.436246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.436341 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.436872 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:32.436881 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:32.438459 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:32.439838 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:32.439843 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:32.439845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:32.440055 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:32.440217 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:32.440217 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:32.440218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:32.440305 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:32.440501 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:32.440505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:32.440608 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.440685 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.440793 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260926 13:18:32.441438 20025 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40261
I20260926 13:18:32.441534 20025 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:32.441706 20025 heartbeater.cc:511] Master 127.0.98.62:40261 requested a full tablet report, sending...
I20260926 13:18:32.442060 19566 ts_manager.cc:194] Registered new tserver with Master: 3a68154f26cc43059f42e3ff5a716091 (127.0.98.3:44969)
I20260926 13:18:32.442458 19566 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46083
I20260926 13:18:32.448462 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:18:32.456265 19605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.449773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.0.98.62:40261)
0926 13:18:32.449822 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.449824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.449943 (+ 119us) server_negotiation.cc:407] Connection header received
0926 13:18:32.450021 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.450022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.450031 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.450086 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.450694 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.451193 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.451951 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.452158 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.452236 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.453651 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.453653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.453653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:32.453659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.455255 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.455585 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.455585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.455586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.455632 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:32.455827 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:32.455828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:32.455828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:32.455962 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.456035 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.456123 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.456183 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 13:18:32.457798 19566 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:18:32.459438 19566 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:18:32.465847 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.461554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44969 (local address 127.0.0.1:43976)
0926 13:18:32.461961 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.461992 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.462071 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.462413 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.462416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.462417 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:32.462477 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.462478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.463424 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.463425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.464586 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.464589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.465494 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.465496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.465586 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.465601 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.465711 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.465769 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260926 13:18:32.465860 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.461753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.0.98.3:44969)
0926 13:18:32.461935 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.461939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.462053 (+ 114us) server_negotiation.cc:407] Connection header received
0926 13:18:32.462179 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.462184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.462201 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.462332 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:32.462553 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.463329 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.464678 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.465378 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.465495 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.465559 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.465735 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.465774 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260926 13:18:32.466008 20035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.461581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.0.1:40808)
0926 13:18:32.462006 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.462017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.462087 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.462417 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.462418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.462420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:32.462507 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.462509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.463819 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.463820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.464862 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.464864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.465751 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.465752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.465825 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.465829 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.465880 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.465921 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":51,"threads_started":1}
I20260926 13:18:32.466274 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.461808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.0.98.2:40141)
0926 13:18:32.462025 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.462029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.462071 (+ 42us) server_negotiation.cc:407] Connection header received
0926 13:18:32.462192 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.462195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.462211 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.462305 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:32.462602 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.463706 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.464942 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.465595 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.465663 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.465715 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.466115 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.466196 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260926 13:18:32.466398 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.461809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.0.98.1:32963)
0926 13:18:32.462010 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.462016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.462218 (+ 202us) server_negotiation.cc:407] Connection header received
0926 13:18:32.462291 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.462604 (+ 313us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.462628 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.462762 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:32.462847 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.463879 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.465174 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.465910 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.465980 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.466032 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.466253 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.466305 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":107,"threads_started":1}
I20260926 13:18:32.466566 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.461554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32963 (local address 127.0.0.1:57124)
0926 13:18:32.461712 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.462151 (+ 439us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.462218 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.462724 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.462725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.462727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:32.462770 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.462772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.463978 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.463979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.465081 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.465084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.466066 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.466068 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.466126 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.466130 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.466224 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.466259 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"spinlock_wait_cycles":896,"thread_start_us":57,"threads_started":1}
I20260926 13:18:32.466977 19825 tablet_service.cc:1467] Processing CreateTablet for tablet 57e4a99e81a84919af5e0d277ece8ecb (DEFAULT_TABLE table=test-table [id=fac5b2b43f204e68b962777dc0d2c161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:32.467154 19691 tablet_service.cc:1467] Processing CreateTablet for tablet 57e4a99e81a84919af5e0d277ece8ecb (DEFAULT_TABLE table=test-table [id=fac5b2b43f204e68b962777dc0d2c161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:32.467334 19825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e4a99e81a84919af5e0d277ece8ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:32.467445 19691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e4a99e81a84919af5e0d277ece8ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:32.467342 19959 tablet_service.cc:1467] Processing CreateTablet for tablet 57e4a99e81a84919af5e0d277ece8ecb (DEFAULT_TABLE table=test-table [id=fac5b2b43f204e68b962777dc0d2c161]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:32.467694 19959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57e4a99e81a84919af5e0d277ece8ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:32.470573 20039 tablet_bootstrap.cc:492] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Bootstrap starting.
I20260926 13:18:32.470849 20041 tablet_bootstrap.cc:492] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732: Bootstrap starting.
I20260926 13:18:32.471204 20039 tablet_bootstrap.cc:654] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:32.471412 20041 tablet_bootstrap.cc:654] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:32.471722 20039 log.cc:824] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:32.471903 20041 log.cc:824] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:32.472226 20040 tablet_bootstrap.cc:492] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: Bootstrap starting.
I20260926 13:18:32.472506 20039 tablet_bootstrap.cc:492] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: No bootstrap required, opened a new log
I20260926 13:18:32.472585 20039 ts_tablet_manager.cc:1397] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:32.472586 20041 tablet_bootstrap.cc:492] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732: No bootstrap required, opened a new log
I20260926 13:18:32.472653 20041 ts_tablet_manager.cc:1397] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:32.472837 20040 tablet_bootstrap.cc:654] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:32.473374 20040 log.cc:824] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:32.474129 20040 tablet_bootstrap.cc:492] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: No bootstrap required, opened a new log
I20260926 13:18:32.474217 20040 ts_tablet_manager.cc:1397] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:32.474630 20041 raft_consensus.cc:348] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.474784 20041 raft_consensus.cc:374] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:32.474685 20039 raft_consensus.cc:348] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.474829 20041 raft_consensus.cc:729] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 566949a4e15a4504aa4c6767184c8732, State: Initialized, Role: FOLLOWER
I20260926 13:18:32.474840 20039 raft_consensus.cc:374] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:32.474871 20039 raft_consensus.cc:729] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a68154f26cc43059f42e3ff5a716091, State: Initialized, Role: FOLLOWER
I20260926 13:18:32.474946 20041 consensus_queue.cc:260] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.474999 20039 consensus_queue.cc:260] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.475221 20041 ts_tablet_manager.cc:1428] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:32.475278 20025 heartbeater.cc:503] Master 127.0.98.62:40261 was elected leader, sending a full tablet report...
I20260926 13:18:32.475353 19757 heartbeater.cc:503] Master 127.0.98.62:40261 was elected leader, sending a full tablet report...
I20260926 13:18:32.475513 20039 ts_tablet_manager.cc:1428] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 13:18:32.476212 20040 raft_consensus.cc:348] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.476423 20040 raft_consensus.cc:374] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:32.476470 20040 raft_consensus.cc:729] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48b4d23c985f4f9a977335d4b8056726, State: Initialized, Role: FOLLOWER
I20260926 13:18:32.476596 20040 consensus_queue.cc:260] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.476861 20040 ts_tablet_manager.cc:1428] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:18:32.476871 19891 heartbeater.cc:503] Master 127.0.98.62:40261 was elected leader, sending a full tablet report...
I20260926 13:18:32.516041 20047 raft_consensus.cc:482] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:32.516156 20047 raft_consensus.cc:504] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.516501 20047 leader_election.cc:288] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a68154f26cc43059f42e3ff5a716091 (127.0.98.3:44969), 566949a4e15a4504aa4c6767184c8732 (127.0.98.1:32963)
I20260926 13:18:32.519663 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.516765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50697 (local address 127.0.98.1:32963)
0926 13:18:32.516813 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.516815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.516852 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:18:32.516875 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.516876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.516886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.516962 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:32.517155 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.517739 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.518791 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.519322 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.519383 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.519394 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.519559 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.519593 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:18:32.519663 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.516672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32963 (local address 127.0.98.2:50697)
0926 13:18:32.516742 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.516755 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.516806 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.516971 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.516972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.516974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:32.517052 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.517054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.517810 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.517811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.518706 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.518708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.519428 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.519429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.519487 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.519490 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.519554 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.519591 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":31}
I20260926 13:18:32.519860 20036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.516728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52563 (local address 127.0.98.3:44969)
0926 13:18:32.516800 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.516802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.517019 (+ 217us) server_negotiation.cc:407] Connection header received
0926 13:18:32.517064 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.517065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.517075 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.517132 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:32.517296 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.517814 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.518884 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.519475 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.519552 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.519562 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.519716 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.519769 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:18:32.520059 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e4a99e81a84919af5e0d277ece8ecb" candidate_uuid: "48b4d23c985f4f9a977335d4b8056726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "566949a4e15a4504aa4c6767184c8732" is_pre_election: true
I20260926 13:18:32.520145 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.516673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44969 (local address 127.0.98.2:52563)
0926 13:18:32.516962 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.516978 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.517022 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.517154 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.517155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.517156 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:32.517229 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.517231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.517883 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.517884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.518799 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.518801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.519564 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.519565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.519644 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.519647 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.520056 (+ 409us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.520090 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":96,"thread_start_us":131,"threads_started":1}
I20260926 13:18:32.520216 19711 raft_consensus.cc:2454] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b4d23c985f4f9a977335d4b8056726 in term 0.
I20260926 13:18:32.520444 19780 leader_election.cc:302] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48b4d23c985f4f9a977335d4b8056726, 566949a4e15a4504aa4c6767184c8732; no voters:
I20260926 13:18:32.520462 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e4a99e81a84919af5e0d277ece8ecb" candidate_uuid: "48b4d23c985f4f9a977335d4b8056726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a68154f26cc43059f42e3ff5a716091" is_pre_election: true
I20260926 13:18:32.520591 19979 raft_consensus.cc:2454] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48b4d23c985f4f9a977335d4b8056726 in term 0.
I20260926 13:18:32.520659 20047 raft_consensus.cc:2781] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:18:32.520740 20047 raft_consensus.cc:482] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:18:32.520768 20047 raft_consensus.cc:3037] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:32.521471 20047 raft_consensus.cc:504] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.521621 20047 leader_election.cc:288] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [CANDIDATE]: Term 1 election: Requested vote from peers 3a68154f26cc43059f42e3ff5a716091 (127.0.98.3:44969), 566949a4e15a4504aa4c6767184c8732 (127.0.98.1:32963)
I20260926 13:18:32.521754 19979 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e4a99e81a84919af5e0d277ece8ecb" candidate_uuid: "48b4d23c985f4f9a977335d4b8056726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a68154f26cc43059f42e3ff5a716091"
I20260926 13:18:32.521772 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57e4a99e81a84919af5e0d277ece8ecb" candidate_uuid: "48b4d23c985f4f9a977335d4b8056726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "566949a4e15a4504aa4c6767184c8732"
I20260926 13:18:32.521832 19979 raft_consensus.cc:3037] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:32.521910 19711 raft_consensus.cc:3037] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:32.522625 19979 raft_consensus.cc:2454] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b4d23c985f4f9a977335d4b8056726 in term 1.
I20260926 13:18:32.522722 19711 raft_consensus.cc:2454] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48b4d23c985f4f9a977335d4b8056726 in term 1.
I20260926 13:18:32.522799 19781 leader_election.cc:302] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a68154f26cc43059f42e3ff5a716091, 48b4d23c985f4f9a977335d4b8056726; no voters:
I20260926 13:18:32.522900 20047 raft_consensus.cc:2781] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:32.523065 20047 raft_consensus.cc:686] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 1 LEADER]: Becoming Leader. State: Replica: 48b4d23c985f4f9a977335d4b8056726, State: Running, Role: LEADER
I20260926 13:18:32.523157 20047 consensus_queue.cc:237] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } }
I20260926 13:18:32.524014 19566 catalog_manager.cc:5495] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48b4d23c985f4f9a977335d4b8056726 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "48b4d23c985f4f9a977335d4b8056726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48b4d23c985f4f9a977335d4b8056726" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40141 } health_report { overall_health: HEALTHY } } }
W20260926 13:18:32.527427 19892 tablet.cc:2369] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:18:32.538520 20037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.536051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.98.2:40141)
0926 13:18:32.536114 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.536116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.536124 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:18:32.536161 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.536161 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.536187 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.536272 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:18:32.536416 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.536887 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.537654 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.537862 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.537939 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.538102 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:18:32.538271 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:18:32.538376 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.538394 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.538414 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 13:18:32.540307 19979 raft_consensus.cc:1261] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 1 FOLLOWER]: Refusing update from remote peer 48b4d23c985f4f9a977335d4b8056726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:32.540309 19711 raft_consensus.cc:1261] T 57e4a99e81a84919af5e0d277ece8ecb P 566949a4e15a4504aa4c6767184c8732 [term 1 FOLLOWER]: Refusing update from remote peer 48b4d23c985f4f9a977335d4b8056726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:32.540701 20047 consensus_queue.cc:1035] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "566949a4e15a4504aa4c6767184c8732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:32.540859 20049 consensus_queue.cc:1035] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a68154f26cc43059f42e3ff5a716091" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:32.543107 20052 mvcc.cc:205] Tried to move back new op lower bound from 7333596006562418688 to 7333596006495641600. Current Snapshot: MvccSnapshot[applied={T|T < 7333596006562418688}]
I20260926 13:18:32.543217 20054 mvcc.cc:205] Tried to move back new op lower bound from 7333596006562418688 to 7333596006495641600. Current Snapshot: MvccSnapshot[applied={T|T < 7333596006562418688}]
I20260926 13:18:32.543370 20053 mvcc.cc:205] Tried to move back new op lower bound from 7333596006562418688 to 7333596006495641600. Current Snapshot: MvccSnapshot[applied={T|T < 7333596006562418688}]
I20260926 13:18:32.545482 19566 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:32.545583 19566 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:32.546366 19566 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8467a06a338f400b9ff35cc257c76d65: Sending DeleteTablet for 3 replicas of tablet 57e4a99e81a84919af5e0d277ece8ecb
I20260926 13:18:32.546702 19825 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e4a99e81a84919af5e0d277ece8ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40808
I20260926 13:18:32.546698 19691 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e4a99e81a84919af5e0d277ece8ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57124
I20260926 13:18:32.546711 19959 tablet_service.cc:1514] Processing DeleteTablet for tablet 57e4a99e81a84919af5e0d277ece8ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43976
I20260926 13:18:32.546942 20064 tablet_replica.cc:330] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: stopping tablet replica
I20260926 13:18:32.547041 20064 raft_consensus.cc:2229] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:32.547142 20064 raft_consensus.cc:2258] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:32.547305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19628
I20260926 13:18:32.548193 20064 ts_tablet_manager.cc:1905] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:32.550057 20064 ts_tablet_manager.cc:1918] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:32.550133 20064 log.cc:1196] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/57e4a99e81a84919af5e0d277ece8ecb
I20260926 13:18:32.550423 20064 ts_tablet_manager.cc:1939] T 57e4a99e81a84919af5e0d277ece8ecb P 3a68154f26cc43059f42e3ff5a716091: Deleting consensus metadata
I20260926 13:18:32.550863 19553 catalog_manager.cc:4841] TS 3a68154f26cc43059f42e3ff5a716091 (127.0.98.3:44969): tablet 57e4a99e81a84919af5e0d277ece8ecb (table test-table [id=fac5b2b43f204e68b962777dc0d2c161]) successfully deleted
W20260926 13:18:32.553524 19552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32963 (error 108)
I20260926 13:18:32.553758 20062 tablet_replica.cc:330] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726: stopping tablet replica
I20260926 13:18:32.553865 20062 raft_consensus.cc:2229] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:18:32.553960 20062 raft_consensus.cc:2258] T 57e4a99e81a84919af5e0d277ece8ecb P 48b4d23c985f4f9a977335d4b8056726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:32.554065 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.553929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32963 (local address 127.0.0.1:57132)
0926 13:18:32.553991 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.554033 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:18:32.554138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19762
W20260926 13:18:32.554260 19552 catalog_manager.cc:4584] TS 566949a4e15a4504aa4c6767184c8732 (127.0.98.1:32963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 57e4a99e81a84919af5e0d277ece8ecb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32963: connect: Connection refused (error 111)
I20260926 13:18:32.558784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19896
I20260926 13:18:32.558948 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.558783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40141 (local address 127.0.0.1:40828)
0926 13:18:32.558882 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.558913 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 13:18:32.563347 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 19534
2026-09-26T13:18:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1882 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T13:18:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:32Z Disabled control of system clock
I20260926 13:18:32.590090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45025
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:32.670296 20069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:32.670506 20069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:32.670527 20069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:32.671594 20069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:32.671626 20069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:32.671639 20069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:32.671651 20069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:32.671663 20069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:32.672907 20069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:32.673094 20069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:32.673313 20069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:18:32.678074 20069 server_base.cc:1034] running on GCE node
W20260926 13:18:32.678085 20074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.678093 20075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.678292 20077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:32.678552 20069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:32.678865 20069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:32.680051 20069 hybrid_clock.cc:648] HybridClock initialized: now 1790428712680029 us; error 37 us; skew 500 ppm
I20260926 13:18:32.681429 20069 webserver.cc:460] Webserver started at http://127.0.98.62:40933/ using document root <none> and password file <none>
I20260926 13:18:32.681711 20069 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:32.681795 20069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:32.681982 20069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:32.683192 20069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "856e1ca561124a4393a46379b7654fed"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "6e4a99129d1b6c9f929926c01b36cb8f"
server_key_iv: "ca958b19bc85924e980a2c361b6e2280"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.683616 20069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "856e1ca561124a4393a46379b7654fed"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "6e4a99129d1b6c9f929926c01b36cb8f"
server_key_iv: "ca958b19bc85924e980a2c361b6e2280"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.685073 20069 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:18:32.685844 20083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.686043 20069 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:32.686156 20069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "856e1ca561124a4393a46379b7654fed"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "6e4a99129d1b6c9f929926c01b36cb8f"
server_key_iv: "ca958b19bc85924e980a2c361b6e2280"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.686295 20069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:32.695860 20069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:32.696161 20069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:32.696336 20069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:32.700696 20069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45025
I20260926 13:18:32.700722 20135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45025 every 8 connection(s)
I20260926 13:18:32.701079 20069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:32.701999 20136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:32.704118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20069
I20260926 13:18:32.704268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:32.704283 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed: Bootstrap starting.
I20260926 13:18:32.704510 392 external_mini_cluster.cc:1428] Setting key 4460b338b73146b5b8b30cea311ce1a5
I20260926 13:18:32.704932 20136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:32.705292 20136 log.cc:824] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:32.706037 20136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed: No bootstrap required, opened a new log
I20260926 13:18:32.708313 20136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856e1ca561124a4393a46379b7654fed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260926 13:18:32.708453 20136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:32.708477 20136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856e1ca561124a4393a46379b7654fed, State: Initialized, Role: FOLLOWER
I20260926 13:18:32.708611 20136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856e1ca561124a4393a46379b7654fed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260926 13:18:32.708693 20136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:32.708725 20136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:32.708763 20136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:32.709456 20136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856e1ca561124a4393a46379b7654fed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260926 13:18:32.709559 20136 leader_election.cc:302] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 856e1ca561124a4393a46379b7654fed; no voters:
I20260926 13:18:32.709725 20136 leader_election.cc:288] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:32.709784 20141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:32.709980 20141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [term 1 LEADER]: Becoming Leader. State: Replica: 856e1ca561124a4393a46379b7654fed, State: Running, Role: LEADER
I20260926 13:18:32.710266 20141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856e1ca561124a4393a46379b7654fed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } }
I20260926 13:18:32.710039 20136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:32.710652 20142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 856e1ca561124a4393a46379b7654fed. Latest consensus state: current_term: 1 leader_uuid: "856e1ca561124a4393a46379b7654fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856e1ca561124a4393a46379b7654fed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } } }
I20260926 13:18:32.710661 20143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "856e1ca561124a4393a46379b7654fed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "856e1ca561124a4393a46379b7654fed" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45025 } } }
I20260926 13:18:32.710744 20142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:32.710750 20143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:32.711150 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.705080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.0.98.62:45025)
0926 13:18:32.705296 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.705305 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.705337 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:18:32.705875 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.705884 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.705968 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.706118 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.706428 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.707811 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.708666 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.709273 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.709401 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.710186 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.710189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.710190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:32.710384 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.710435 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:32.710449 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.710541 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.710801 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.710880 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260926 13:18:32.710980 20145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:32.711753 20145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:32.713227 20145 catalog_manager.cc:1347] Generated new cluster ID: c89e5a5bb26349f3a8c2f5ce6ab944d3
I20260926 13:18:32.713270 20145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:32.727807 20145 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:32.728286 20145 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:32.744716 20145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed: Generated new TSK 0
I20260926 13:18:32.744903 20145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:32.748028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:32.828351 20160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:32.828550 20160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:32.828572 20160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:32.829742 20160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:32.829784 20160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:32.829808 20160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 13:18:32.831210 20160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:32.831454 20160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:32.831764 20160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:32.836807 20168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.836814 20165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.836884 20166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:32.837042 20160 server_base.cc:1034] running on GCE node
I20260926 13:18:32.837227 20160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:32.837420 20160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:32.838567 20160 hybrid_clock.cc:648] HybridClock initialized: now 1790428712838554 us; error 27 us; skew 500 ppm
I20260926 13:18:32.839802 20160 webserver.cc:460] Webserver started at http://127.0.98.1:42537/ using document root <none> and password file <none>
I20260926 13:18:32.840051 20160 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:32.840099 20160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:32.840257 20160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:32.841323 20160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc07cee217604d92969322c36c225037"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "2fc093c5ae61fcedbe3848858ec67e06"
server_key_iv: "c90a4c12300f65f86c7ee3d9d7015a97"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.841675 20160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc07cee217604d92969322c36c225037"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "2fc093c5ae61fcedbe3848858ec67e06"
server_key_iv: "c90a4c12300f65f86c7ee3d9d7015a97"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.843039 20160 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:32.843751 20174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.843935 20160 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:32.844029 20160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal
uuid: "cc07cee217604d92969322c36c225037"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "2fc093c5ae61fcedbe3848858ec67e06"
server_key_iv: "c90a4c12300f65f86c7ee3d9d7015a97"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.844110 20160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:32.864113 20160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:32.864423 20160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:32.864567 20160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:32.864789 20160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:32.865135 20160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:32.865185 20160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.865224 20160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:32.865252 20160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.870695 20160 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44391
I20260926 13:18:32.870743 20287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44391 every 8 connection(s)
I20260926 13:18:32.871057 20160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/data/info.pb
I20260926 13:18:32.872905 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20160
I20260926 13:18:32.873003 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-0/wal/instance
I20260926 13:18:32.873199 392 external_mini_cluster.cc:1428] Setting key 05eab9ef844bd6c7941262afa4ec542c
I20260926 13:18:32.874189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 13:18:32.876595 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.871717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59285 (local address 127.0.98.62:45025)
0926 13:18:32.871765 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.871768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.871976 (+ 208us) server_negotiation.cc:407] Connection header received
0926 13:18:32.872509 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.872511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.872525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.872586 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.873411 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.874058 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.875128 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.875340 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.875420 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.876311 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.876312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.876313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:32.876318 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.876344 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:32.876348 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.876403 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.876497 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.876521 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:18:32.876820 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.871674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.1:59285)
0926 13:18:32.871852 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.871885 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.872399 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.872672 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.872676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.872687 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:32.873291 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.873297 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.874147 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.874149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.875035 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.875037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.875429 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.875430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.875573 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.876143 (+ 570us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:32.876143 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:32.876171 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:32.876193 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:32.876419 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:32.876424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:32.876427 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.876496 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.876597 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260926 13:18:32.877450 20288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45025
I20260926 13:18:32.877542 20288 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:32.877729 20288 heartbeater.cc:511] Master 127.0.98.62:45025 requested a full tablet report, sending...
I20260926 13:18:32.878330 20100 ts_manager.cc:194] Registered new tserver with Master: cc07cee217604d92969322c36c225037 (127.0.98.1:44391)
I20260926 13:18:32.879290 20100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59285
W20260926 13:18:32.954547 20291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:32.954739 20291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:32.954761 20291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:32.955834 20291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:32.955866 20291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:32.955891 20291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 13:18:32.957149 20291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:32.957325 20291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:32.957567 20291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:32.962438 20297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.962502 20296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:32.962442 20299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:32.962481 20291 server_base.cc:1034] running on GCE node
I20260926 13:18:32.962857 20291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:32.963052 20291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:32.964190 20291 hybrid_clock.cc:648] HybridClock initialized: now 1790428712964175 us; error 30 us; skew 500 ppm
I20260926 13:18:32.965292 20291 webserver.cc:460] Webserver started at http://127.0.98.2:44561/ using document root <none> and password file <none>
I20260926 13:18:32.965521 20291 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:32.965564 20291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:32.965652 20291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:32.966728 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data/instance:
uuid: "f29f3da9960b432c95016d91bdbe326e"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "aac305021189e188c2b40e196bfa0fd9"
server_key_iv: "414b38493bcc60d0fb54fd3d7a224dd7"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.967024 20291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f29f3da9960b432c95016d91bdbe326e"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "aac305021189e188c2b40e196bfa0fd9"
server_key_iv: "414b38493bcc60d0fb54fd3d7a224dd7"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.968194 20291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:32.968905 20305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.969074 20291 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:32.969152 20291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal
uuid: "f29f3da9960b432c95016d91bdbe326e"
format_stamp: "Formatted at 2026-09-26 13:18:32 on dist-test-slave-f864"
server_key: "aac305021189e188c2b40e196bfa0fd9"
server_key_iv: "414b38493bcc60d0fb54fd3d7a224dd7"
server_key_version: "encryptionkey@0"
I20260926 13:18:32.969205 20291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:32.984532 20291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:32.984805 20291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:32.984910 20291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:32.985101 20291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:32.985448 20291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:32.985499 20291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.985544 20291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:32.985571 20291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:32.991369 20291 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45179
I20260926 13:18:32.991457 20418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45179 every 8 connection(s)
I20260926 13:18:32.991777 20291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/data/info.pb
I20260926 13:18:32.997195 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.992700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39903 (local address 127.0.98.62:45025)
0926 13:18:32.992762 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:32.992764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:32.993028 (+ 264us) server_negotiation.cc:407] Connection header received
0926 13:18:32.993584 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:32.993585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:32.993599 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:32.993649 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:32.994336 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.994829 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.995845 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.996043 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.996121 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.996904 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:32.996906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:32.996906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:32.996911 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:32.996927 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:32.996930 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:32.996998 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:32.997090 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:18:32.997116 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 13:18:32.997452 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:32.992682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.2:39903)
0926 13:18:32.992904 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:32.992936 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:32.993463 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:32.993727 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:32.993730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:32.993739 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:32.994236 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.994241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.994926 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.994928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.995729 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:32.995731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:32.996130 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:32.996132 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:32.996266 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:32.996766 (+ 500us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:32.996766 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:32.996794 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:32.996816 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:32.996998 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:32.997004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:32.997008 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:32.997094 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:18:32.997195 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":100,"threads_started":1}
I20260926 13:18:32.997965 20419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45025
I20260926 13:18:32.998072 20419 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:32.998291 20419 heartbeater.cc:511] Master 127.0.98.62:45025 requested a full tablet report, sending...
I20260926 13:18:32.998675 20100 ts_manager.cc:194] Registered new tserver with Master: f29f3da9960b432c95016d91bdbe326e (127.0.98.2:45179)
I20260926 13:18:32.999050 20100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39903
I20260926 13:18:32.999202 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20291
I20260926 13:18:32.999357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-1/wal/instance
I20260926 13:18:32.999583 392 external_mini_cluster.cc:1428] Setting key 80e92f283ba3cba2e89e243341d025f3
I20260926 13:18:33.000464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--builtin_ntp_servers=127.0.98.20:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 13:18:33.081035 20422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:33.081259 20422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:33.081295 20422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:33.082559 20422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:33.082615 20422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:33.082656 20422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 13:18:33.084043 20422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:33.084313 20422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:33.084602 20422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:33.089521 20428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:33.089545 20430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:33.089573 20422 server_base.cc:1034] running on GCE node
W20260926 13:18:33.089668 20427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:33.089922 20422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:33.090152 20422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:33.091361 20422 hybrid_clock.cc:648] HybridClock initialized: now 1790428713091340 us; error 33 us; skew 500 ppm
I20260926 13:18:33.092648 20422 webserver.cc:460] Webserver started at http://127.0.98.3:38527/ using document root <none> and password file <none>
I20260926 13:18:33.092916 20422 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:33.092962 20422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:33.093134 20422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:33.094254 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data/instance:
uuid: "c5ed975c2f014a55afc0145bbf35312b"
format_stamp: "Formatted at 2026-09-26 13:18:33 on dist-test-slave-f864"
server_key: "bf740bccdf36e72b100cb6df79c3ffbd"
server_key_iv: "9caf55480136d24d30e4c2dc246534fc"
server_key_version: "encryptionkey@0"
I20260926 13:18:33.094624 20422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c5ed975c2f014a55afc0145bbf35312b"
format_stamp: "Formatted at 2026-09-26 13:18:33 on dist-test-slave-f864"
server_key: "bf740bccdf36e72b100cb6df79c3ffbd"
server_key_iv: "9caf55480136d24d30e4c2dc246534fc"
server_key_version: "encryptionkey@0"
I20260926 13:18:33.095862 20422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:33.096544 20436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:33.096691 20422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:33.096767 20422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal
uuid: "c5ed975c2f014a55afc0145bbf35312b"
format_stamp: "Formatted at 2026-09-26 13:18:33 on dist-test-slave-f864"
server_key: "bf740bccdf36e72b100cb6df79c3ffbd"
server_key_iv: "9caf55480136d24d30e4c2dc246534fc"
server_key_version: "encryptionkey@0"
I20260926 13:18:33.096817 20422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:33.113097 20422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:33.113344 20422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:33.113442 20422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:33.113627 20422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 13:18:33.113888 20422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 13:18:33.113925 20422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:33.113947 20422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 13:18:33.113960 20422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:33.119432 20422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34477
I20260926 13:18:33.119493 20549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34477 every 8 connection(s)
I20260926 13:18:33.119791 20422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/data/info.pb
I20260926 13:18:33.124858 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.120535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35447 (local address 127.0.98.62:45025)
0926 13:18:33.120583 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.120586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.120869 (+ 283us) server_negotiation.cc:407] Connection header received
0926 13:18:33.121351 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.121352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.121362 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.121413 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:33.122086 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.122580 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.123524 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.123695 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.123742 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.124564 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:33.124565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:33.124566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:33.124570 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:33.124583 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:33.124586 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:33.124629 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.124726 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.124758 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:18:33.125106 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.120540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45025 (local address 127.0.98.3:35447)
0926 13:18:33.120748 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.120785 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.121269 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.121507 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:33.121512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:33.121523 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:33.122018 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.122022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.122662 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.122663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.123451 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.123453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.123814 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.123816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.123942 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.124435 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 13:18:33.124436 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:33.124463 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 13:18:33.124487 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:33.124640 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:33.124644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:33.124646 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 13:18:33.124697 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:18:33.124849 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260926 13:18:33.125093 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20422
I20260926 13:18:33.125205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal/instance
I20260926 13:18:33.125460 392 external_mini_cluster.cc:1428] Setting key 955e21e6f51ccd013a269cf553e9d597
I20260926 13:18:33.125663 20550 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45025
I20260926 13:18:33.125757 20550 heartbeater.cc:464] Registering TS with master...
I20260926 13:18:33.125967 20550 heartbeater.cc:511] Master 127.0.98.62:45025 requested a full tablet report, sending...
I20260926 13:18:33.126416 20099 ts_manager.cc:194] Registered new tserver with Master: c5ed975c2f014a55afc0145bbf35312b (127.0.98.3:34477)
I20260926 13:18:33.126808 20099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35447
I20260926 13:18:33.137398 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 13:18:33.141630 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.138746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.0.98.62:45025)
0926 13:18:33.138821 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.138824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.138935 (+ 111us) server_negotiation.cc:407] Connection header received
0926 13:18:33.139009 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.139010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.139020 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.139084 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:33.139260 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.139734 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.140484 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.140688 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.140758 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.141374 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:33.141375 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:33.141376 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:33.141380 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:33.141393 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:33.141396 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:33.141471 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.141530 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.141550 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 13:18:33.143128 20099 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 13:18:33.144537 20099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 13:18:33.151100 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.146655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44391 (local address 127.0.0.1:44822)
0926 13:18:33.146998 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.147007 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.147065 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.148037 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:33.148037 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:33.148039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:33.148080 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.148082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.148921 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.148922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.149986 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.149989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.150903 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.150903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.150957 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.150967 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 13:18:33.151014 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 13:18:33.151048 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":85,"thread_start_us":47,"threads_started":1}
I20260926 13:18:33.151170 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.146898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44822 (local address 127.0.98.1:44391)
0926 13:18:33.147880 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.147884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.147895 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:18:33.147930 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.147932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.147945 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.148038 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:33.148147 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.148824 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.150076 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.150820 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.150897 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.150999 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.151027 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.151066 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":83,"threads_started":1}
I20260926 13:18:33.151851 20222 tablet_service.cc:1467] Processing CreateTablet for tablet c13a9c3cfb104e8981e7701c790fce45 (DEFAULT_TABLE table=test-table [id=d2ca32bbc3ea4b1cb2a10a0aa521d225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:33.152233 20222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13a9c3cfb104e8981e7701c790fce45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:33.152271 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.146661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34477 (local address 127.0.0.1:37400)
0926 13:18:33.147155 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.147163 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.147198 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.147777 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:33.147778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:33.147780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:33.147835 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.147837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.149053 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.149054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.150299 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.150301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.152024 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.152025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.152084 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.152087 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:33.152151 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 13:18:33.152195 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":100,"threads_started":1}
I20260926 13:18:33.152252 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.146864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.0.98.3:34477)
0926 13:18:33.147632 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.147635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.147642 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:18:33.147668 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.147670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.147680 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.147759 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:33.147919 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.148977 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.151288 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.151937 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.151991 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.152075 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.152169 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.152196 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":82,"threads_started":1}
I20260926 13:18:33.153081 20484 tablet_service.cc:1467] Processing CreateTablet for tablet c13a9c3cfb104e8981e7701c790fce45 (DEFAULT_TABLE table=test-table [id=d2ca32bbc3ea4b1cb2a10a0aa521d225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:33.153445 20484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13a9c3cfb104e8981e7701c790fce45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:33.154381 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.146832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.98.2:45179)
0926 13:18:33.147341 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.147345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.147371 (+ 26us) server_negotiation.cc:407] Connection header received
0926 13:18:33.147412 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.147414 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.147431 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.147516 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:33.147712 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.148784 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.150395 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.151086 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.151153 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.151251 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.154248 (+ 2997us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.154294 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":80,"threads_started":1}
I20260926 13:18:33.154592 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.146655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45179 (local address 127.0.0.1:52894)
0926 13:18:33.146806 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.146855 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.146919 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.147553 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:33.147556 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:33.147558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:33.147620 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.147621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.148886 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.148887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.150298 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.150301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.151177 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.151178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.151349 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.151352 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 13:18:33.154472 (+ 3120us) client_negotiation.cc:240] Negotiation successful
0926 13:18:33.154524 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":105,"thread_start_us":99,"threads_started":1}
I20260926 13:18:33.155635 20564 tablet_bootstrap.cc:492] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Bootstrap starting.
I20260926 13:18:33.156206 20564 tablet_bootstrap.cc:654] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:33.156481 20564 log.cc:824] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:33.156581 20565 tablet_bootstrap.cc:492] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Bootstrap starting.
I20260926 13:18:33.157212 20565 tablet_bootstrap.cc:654] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:33.157178 20353 tablet_service.cc:1467] Processing CreateTablet for tablet c13a9c3cfb104e8981e7701c790fce45 (DEFAULT_TABLE table=test-table [id=d2ca32bbc3ea4b1cb2a10a0aa521d225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 13:18:33.157469 20564 tablet_bootstrap.cc:492] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: No bootstrap required, opened a new log
I20260926 13:18:33.157492 20353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c13a9c3cfb104e8981e7701c790fce45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:33.157541 20564 ts_tablet_manager.cc:1397] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:33.157539 20565 log.cc:824] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:33.158200 20565 tablet_bootstrap.cc:492] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: No bootstrap required, opened a new log
I20260926 13:18:33.158284 20565 ts_tablet_manager.cc:1397] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 13:18:33.160084 20564 raft_consensus.cc:348] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.160142 20565 raft_consensus.cc:348] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.160262 20564 raft_consensus.cc:374] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:33.160282 20565 raft_consensus.cc:374] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:33.160300 20564 raft_consensus.cc:729] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc07cee217604d92969322c36c225037, State: Initialized, Role: FOLLOWER
I20260926 13:18:33.160323 20565 raft_consensus.cc:729] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ed975c2f014a55afc0145bbf35312b, State: Initialized, Role: FOLLOWER
I20260926 13:18:33.160434 20564 consensus_queue.cc:260] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.160488 20565 consensus_queue.cc:260] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.160702 20564 ts_tablet_manager.cc:1428] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 13:18:33.160796 20565 ts_tablet_manager.cc:1428] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 13:18:33.160918 20550 heartbeater.cc:503] Master 127.0.98.62:45025 was elected leader, sending a full tablet report...
I20260926 13:18:33.160921 20288 heartbeater.cc:503] Master 127.0.98.62:45025 was elected leader, sending a full tablet report...
I20260926 13:18:33.161232 20569 tablet_bootstrap.cc:492] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: Bootstrap starting.
I20260926 13:18:33.161971 20569 tablet_bootstrap.cc:654] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:33.162350 20569 log.cc:824] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:33.163002 20569 tablet_bootstrap.cc:492] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: No bootstrap required, opened a new log
I20260926 13:18:33.163053 20569 ts_tablet_manager.cc:1397] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 13:18:33.164748 20569 raft_consensus.cc:348] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.164877 20569 raft_consensus.cc:374] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:33.164924 20569 raft_consensus.cc:729] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f29f3da9960b432c95016d91bdbe326e, State: Initialized, Role: FOLLOWER
I20260926 13:18:33.165045 20569 consensus_queue.cc:260] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.165344 20569 ts_tablet_manager.cc:1428] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 13:18:33.165364 20419 heartbeater.cc:503] Master 127.0.98.62:45025 was elected leader, sending a full tablet report...
W20260926 13:18:33.242440 20420 tablet.cc:2369] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:18:33.370385 20551 tablet.cc:2369] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 13:18:33.371685 20289 tablet.cc:2369] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 13:18:33.499483 20570 raft_consensus.cc:482] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 13:18:33.499660 20570 raft_consensus.cc:504] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.500166 20570 leader_election.cc:288] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f29f3da9960b432c95016d91bdbe326e (127.0.98.2:45179), cc07cee217604d92969322c36c225037 (127.0.98.1:44391)
I20260926 13:18:33.503257 20552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.500355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45179 (local address 127.0.98.3:42597)
0926 13:18:33.500426 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.500438 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.500484 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.500667 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:33.500668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:33.500670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:33.500806 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.500809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.501587 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.501589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.502447 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.502450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.503006 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.503007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.503083 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.503087 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:33.503141 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 13:18:33.503180 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 13:18:33.503297 20561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.500403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42597 (local address 127.0.98.2:45179)
0926 13:18:33.500525 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.500527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.500539 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:18:33.500575 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.500577 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.500589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.500651 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:33.500891 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.501506 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.502533 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.502926 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.502982 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.502990 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.503159 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.503207 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 13:18:33.503384 20573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.500386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44391 (local address 127.0.98.3:41155)
0926 13:18:33.500567 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.500577 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.500641 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.500800 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:33.500801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:33.500803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 13:18:33.500895 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.500897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.501664 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.501665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.502551 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:33.502553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.503094 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.503095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:33.503188 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.503192 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 13:18:33.503251 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 13:18:33.503287 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260926 13:18:33.503386 20563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.500441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41155 (local address 127.0.98.1:44391)
0926 13:18:33.500497 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.500499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.500655 (+ 156us) server_negotiation.cc:407] Connection header received
0926 13:18:33.500698 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.500700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.500712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.500768 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 13:18:33.500964 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.501565 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.502639 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.503015 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.503084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.503094 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.503268 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.503317 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 13:18:33.503697 20242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13a9c3cfb104e8981e7701c790fce45" candidate_uuid: "c5ed975c2f014a55afc0145bbf35312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc07cee217604d92969322c36c225037" is_pre_election: true
I20260926 13:18:33.503741 20373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13a9c3cfb104e8981e7701c790fce45" candidate_uuid: "c5ed975c2f014a55afc0145bbf35312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29f3da9960b432c95016d91bdbe326e" is_pre_election: true
I20260926 13:18:33.503895 20373 raft_consensus.cc:2454] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ed975c2f014a55afc0145bbf35312b in term 0.
I20260926 13:18:33.503890 20242 raft_consensus.cc:2454] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ed975c2f014a55afc0145bbf35312b in term 0.
I20260926 13:18:33.504125 20437 leader_election.cc:302] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ed975c2f014a55afc0145bbf35312b, f29f3da9960b432c95016d91bdbe326e; no voters:
I20260926 13:18:33.504289 20570 raft_consensus.cc:2781] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 13:18:33.504356 20570 raft_consensus.cc:482] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 13:18:33.504374 20570 raft_consensus.cc:3037] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:33.505085 20570 raft_consensus.cc:504] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.505228 20570 leader_election.cc:288] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [CANDIDATE]: Term 1 election: Requested vote from peers f29f3da9960b432c95016d91bdbe326e (127.0.98.2:45179), cc07cee217604d92969322c36c225037 (127.0.98.1:44391)
I20260926 13:18:33.505390 20373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13a9c3cfb104e8981e7701c790fce45" candidate_uuid: "c5ed975c2f014a55afc0145bbf35312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f29f3da9960b432c95016d91bdbe326e"
I20260926 13:18:33.505486 20373 raft_consensus.cc:3037] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:33.506211 20373 raft_consensus.cc:2454] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ed975c2f014a55afc0145bbf35312b in term 1.
I20260926 13:18:33.506246 20242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c13a9c3cfb104e8981e7701c790fce45" candidate_uuid: "c5ed975c2f014a55afc0145bbf35312b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc07cee217604d92969322c36c225037"
I20260926 13:18:33.506350 20242 raft_consensus.cc:3037] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:33.506381 20437 leader_election.cc:302] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ed975c2f014a55afc0145bbf35312b, f29f3da9960b432c95016d91bdbe326e; no voters:
I20260926 13:18:33.506476 20570 raft_consensus.cc:2781] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:33.506649 20570 raft_consensus.cc:686] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 1 LEADER]: Becoming Leader. State: Replica: c5ed975c2f014a55afc0145bbf35312b, State: Running, Role: LEADER
I20260926 13:18:33.506749 20570 consensus_queue.cc:237] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } }
I20260926 13:18:33.507467 20242 raft_consensus.cc:2454] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ed975c2f014a55afc0145bbf35312b in term 1.
I20260926 13:18:33.507710 20100 catalog_manager.cc:5495] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ed975c2f014a55afc0145bbf35312b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c5ed975c2f014a55afc0145bbf35312b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ed975c2f014a55afc0145bbf35312b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34477 } health_report { overall_health: HEALTHY } } }
I20260926 13:18:33.592970 20562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.590557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37412 (local address 127.0.98.3:34477)
0926 13:18:33.590624 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:33.590626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:33.590635 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:18:33.590689 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:33.590690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:33.590707 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:33.590787 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:18:33.590934 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.591407 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.592180 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:33.592373 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:33.592457 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:33.592626 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:18:33.592722 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:18:33.592816 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:33.592835 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 13:18:33.592852 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 13:18:33.594810 20242 raft_consensus.cc:1261] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 1 FOLLOWER]: Refusing update from remote peer c5ed975c2f014a55afc0145bbf35312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:33.594903 20373 raft_consensus.cc:1261] T c13a9c3cfb104e8981e7701c790fce45 P f29f3da9960b432c95016d91bdbe326e [term 1 FOLLOWER]: Refusing update from remote peer c5ed975c2f014a55afc0145bbf35312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:33.595290 20570 consensus_queue.cc:1035] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc07cee217604d92969322c36c225037" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:33.595443 20574 consensus_queue.cc:1035] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f29f3da9960b432c95016d91bdbe326e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:33.597746 20577 mvcc.cc:205] Tried to move back new op lower bound from 7333596010881175552 to 7333596010524200960. Current Snapshot: MvccSnapshot[applied={T|T < 7333596010881175552}]
I20260926 13:18:33.598189 20579 mvcc.cc:205] Tried to move back new op lower bound from 7333596010881175552 to 7333596010524200960. Current Snapshot: MvccSnapshot[applied={T|T < 7333596010881175552}]
I20260926 13:18:33.598798 20578 mvcc.cc:205] Tried to move back new op lower bound from 7333596010881175552 to 7333596010524200960. Current Snapshot: MvccSnapshot[applied={T|T < 7333596010881175552}]
I20260926 13:18:33.600764 20100 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:33.600858 20100 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 13:18:33.601589 20100 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 856e1ca561124a4393a46379b7654fed: Sending DeleteTablet for 3 replicas of tablet c13a9c3cfb104e8981e7701c790fce45
I20260926 13:18:33.601933 20484 tablet_service.cc:1514] Processing DeleteTablet for tablet c13a9c3cfb104e8981e7701c790fce45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:33 UTC) from {username='slave'} at 127.0.0.1:37400
I20260926 13:18:33.601936 20222 tablet_service.cc:1514] Processing DeleteTablet for tablet c13a9c3cfb104e8981e7701c790fce45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:33 UTC) from {username='slave'} at 127.0.0.1:44822
I20260926 13:18:33.602154 20587 tablet_replica.cc:330] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: stopping tablet replica
I20260926 13:18:33.602190 20586 tablet_replica.cc:330] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: stopping tablet replica
I20260926 13:18:33.602286 20587 raft_consensus.cc:2229] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 1 LEADER]: Raft consensus shutting down.
I20260926 13:18:33.602348 20353 tablet_service.cc:1514] Processing DeleteTablet for tablet c13a9c3cfb104e8981e7701c790fce45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 13:18:33 UTC) from {username='slave'} at 127.0.0.1:52894
I20260926 13:18:33.602412 20587 raft_consensus.cc:2258] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:33.602397 20586 raft_consensus.cc:2229] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 13:18:33.602610 20586 raft_consensus.cc:2258] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 13:18:33.602957 20587 ts_tablet_manager.cc:1905] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:33.602979 20586 ts_tablet_manager.cc:1905] T c13a9c3cfb104e8981e7701c790fce45 P cc07cee217604d92969322c36c225037: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 13:18:33.602975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20160
I20260926 13:18:33.605315 20587 ts_tablet_manager.cc:1918] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 13:18:33.605413 20587 log.cc:1196] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Deleting WAL directory at /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790428444292953-392-0/minicluster-data/ts-2/wal/wals/c13a9c3cfb104e8981e7701c790fce45
I20260926 13:18:33.605679 20587 ts_tablet_manager.cc:1939] T c13a9c3cfb104e8981e7701c790fce45 P c5ed975c2f014a55afc0145bbf35312b: Deleting consensus metadata
I20260926 13:18:33.606086 20087 catalog_manager.cc:4841] TS c5ed975c2f014a55afc0145bbf35312b (127.0.98.3:34477): tablet c13a9c3cfb104e8981e7701c790fce45 (table test-table [id=d2ca32bbc3ea4b1cb2a10a0aa521d225]) successfully deleted
W20260926 13:18:33.608562 20086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44391 (error 108)
W20260926 13:18:33.609169 20558 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:33.608972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44391 (local address 127.0.0.1:44838)
0926 13:18:33.609024 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.609035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:33.609072 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:33.609132 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44391: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 13:18:33.609303 20086 catalog_manager.cc:4584] TS cc07cee217604d92969322c36c225037 (127.0.98.1:44391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c13a9c3cfb104e8981e7701c790fce45: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44391: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260926 13:18:33.609393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20291
I20260926 13:18:33.615041 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:33.614916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45179 (local address 127.0.0.1:52908)
0926 13:18:33.614972 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:33.615015 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 13:18:33.615042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20422
I20260926 13:18:33.619566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20069
2026-09-26T13:18:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1055 ms)
[----------] 6 tests from EncryptionPolicyTest (8634 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:33 dist-test-slave-f864 krb5kdc[20592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:33 dist-test-slave-f864 krb5kdc[20592](info): set up 2 sockets
Sep 26 13:18:33 dist-test-slave-f864 krb5kdc[20592](info): commencing operation
krb5kdc: starting...
W20260926 13:18:34.666350 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:34 dist-test-slave-f864 krb5kdc[20592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:34.780550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:34.860922 20605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:34.861163 20605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:34.861202 20605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:34.862397 20605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:34.862468 20605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:34.862496 20605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:34.862527 20605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:34.862551 20605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:34.864039 20605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:34.864308 20605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:34.864559 20605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:34.869381 20613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:34.869421 20610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:34.869518 20605 server_base.cc:1034] running on GCE node
W20260926 13:18:34.869393 20611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:34.869798 20605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:34.870033 20605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:34.871217 20605 hybrid_clock.cc:648] HybridClock initialized: now 1790428714871206 us; error 40 us; skew 500 ppm
Sep 26 13:18:34 dist-test-slave-f864 krb5kdc[20592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428714, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:34.878875 20605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:34.879283 20605 webserver.cc:460] Webserver started at http://127.0.0.1:36537/ using document root <none> and password file <none>
I20260926 13:18:34.879503 20605 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:34.879547 20605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:34.879709 20605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:34.880772 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "3c8f63d7729649da903eb8d30458dc2e"
format_stamp: "Formatted at 2026-09-26 13:18:34 on dist-test-slave-f864"
server_key: "6ce0a70841f57ee1f5ad456591e9b85b"
server_key_iv: "ea7c1d0442adb63474d5da956f08e859"
server_key_version: "encryptionkey@0"
I20260926 13:18:34.881134 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c8f63d7729649da903eb8d30458dc2e"
format_stamp: "Formatted at 2026-09-26 13:18:34 on dist-test-slave-f864"
server_key: "6ce0a70841f57ee1f5ad456591e9b85b"
server_key_iv: "ea7c1d0442adb63474d5da956f08e859"
server_key_version: "encryptionkey@0"
I20260926 13:18:34.882488 20605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:34.883203 20620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:34.883406 20605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:34.883495 20605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "3c8f63d7729649da903eb8d30458dc2e"
format_stamp: "Formatted at 2026-09-26 13:18:34 on dist-test-slave-f864"
server_key: "6ce0a70841f57ee1f5ad456591e9b85b"
server_key_iv: "ea7c1d0442adb63474d5da956f08e859"
server_key_version: "encryptionkey@0"
I20260926 13:18:34.883566 20605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:34.894371 20605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:34.896410 20605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:34.896549 20605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:34.900406 20605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44145
I20260926 13:18:34.900457 20672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44145 every 8 connection(s)
I20260926 13:18:34.900799 20605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:34.901604 20673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:34.903882 20673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e: Bootstrap starting.
I20260926 13:18:34.904382 20673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:34.904606 20673 log.cc:824] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:34.905071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20605
I20260926 13:18:34.905189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:34.905326 20673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e: No bootstrap required, opened a new log
I20260926 13:18:34.905414 392 external_mini_cluster.cc:1428] Setting key 46ca8d226bdf54cbdf876f4fbbc39271
I20260926 13:18:34.907778 20673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f63d7729649da903eb8d30458dc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260926 13:18:34.907905 20673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:34.907927 20673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c8f63d7729649da903eb8d30458dc2e, State: Initialized, Role: FOLLOWER
I20260926 13:18:34.908051 20673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f63d7729649da903eb8d30458dc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260926 13:18:34.908123 20673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:34.908145 20673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:34.908211 20673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:34.908728 20673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f63d7729649da903eb8d30458dc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260926 13:18:34.908816 20673 leader_election.cc:302] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c8f63d7729649da903eb8d30458dc2e; no voters:
I20260926 13:18:34.908993 20673 leader_election.cc:288] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:34.909018 20678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:34.909130 20678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [term 1 LEADER]: Becoming Leader. State: Replica: 3c8f63d7729649da903eb8d30458dc2e, State: Running, Role: LEADER
I20260926 13:18:34.909209 20678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f63d7729649da903eb8d30458dc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } }
I20260926 13:18:34.909255 20673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:34.909513 20680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c8f63d7729649da903eb8d30458dc2e. Latest consensus state: current_term: 1 leader_uuid: "3c8f63d7729649da903eb8d30458dc2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f63d7729649da903eb8d30458dc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } } }
I20260926 13:18:34.909729 20680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:34.909674 20679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c8f63d7729649da903eb8d30458dc2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c8f63d7729649da903eb8d30458dc2e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44145 } } }
I20260926 13:18:34.910012 20679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:34.910100 20685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:34.910703 20685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:34.912196 20685 catalog_manager.cc:1347] Generated new cluster ID: 346547fc941c48fe9834d90cac6ac17d
I20260926 13:18:34.912498 20685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:34 dist-test-slave-f864 krb5kdc[20592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:18:34.917542 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:34.905930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37267 (local address 127.0.0.1:44145)
0926 13:18:34.906103 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:34.906112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:34.906155 (+ 43us) server_negotiation.cc:407] Connection header received
0926 13:18:34.906707 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:34.906713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:34.906757 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:34.906865 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:34.907670 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:34.909020 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:34.910129 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:34.910818 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:34.911314 (+ 496us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:34.913703 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:34.913707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:34.913708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:34.913719 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:34.915772 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:34.916193 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:34.916196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:34.916196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:34.916306 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:34.916550 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:34.916551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:34.916551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:34.916870 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:34.917026 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:34.917171 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 13:18:34.917290 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260926 13:18:34.918489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:34.926867 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:34.919833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51545 (local address 127.0.0.1:44145)
0926 13:18:34.919867 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:34.919868 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:34.919998 (+ 130us) server_negotiation.cc:407] Connection header received
0926 13:18:34.920025 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:34.920026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:34.920036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:34.920099 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:34.920666 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:34.921407 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:34.922358 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:34.922710 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:34.922796 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:34.924110 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:34.924112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:34.924112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:34.924119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:34.925662 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:34.926056 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:34.926057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:34.926057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:34.926115 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:34.926365 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:34.926366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:34.926366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:34.926493 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:34.926592 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:34.926707 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 13:18:34.926776 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:18:34.939318 20685 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:34.940062 20685 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:34.944403 20685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c8f63d7729649da903eb8d30458dc2e: Generated new TSK 0
I20260926 13:18:34.944576 20685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:34.949261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20605
W20260926 13:18:34.953451 20590 connection.cc:535] client connection to 127.0.0.1:44145 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44145): Connection reset by peer (error 104)
2026-09-26T13:18:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:35 dist-test-slave-f864 krb5kdc[20704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:35 dist-test-slave-f864 krb5kdc[20704](info): set up 2 sockets
Sep 26 13:18:35 dist-test-slave-f864 krb5kdc[20704](info): commencing operation
krb5kdc: starting...
W20260926 13:18:35.970273 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:36 dist-test-slave-f864 krb5kdc[20704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428716, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:36.088524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:36.167759 20717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:36.167999 20717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:36.168037 20717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:36.169181 20717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:36.169257 20717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:36.169286 20717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:36.169318 20717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:36.169341 20717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:36.170854 20717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:36.171141 20717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:36.171378 20717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:36.176179 20725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:36.176187 20723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:36.176198 20722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:36.176436 20717 server_base.cc:1034] running on GCE node
I20260926 13:18:36.176676 20717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:36.176918 20717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:36.178056 20717 hybrid_clock.cc:648] HybridClock initialized: now 1790428716178031 us; error 41 us; skew 500 ppm
Sep 26 13:18:36 dist-test-slave-f864 krb5kdc[20704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:36.180514 20717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:36.180920 20717 webserver.cc:460] Webserver started at http://127.0.0.1:45737/ using document root <none> and password file <none>
I20260926 13:18:36.181103 20717 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:36.181143 20717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:36.181232 20717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:36.182323 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "7a9464d450e447f78bfa49a9b26bb45a"
format_stamp: "Formatted at 2026-09-26 13:18:36 on dist-test-slave-f864"
server_key: "06846c421cb03b3b3dde1aeeb04dd658"
server_key_iv: "af66834cd5c15211c65e1e24c068d185"
server_key_version: "encryptionkey@0"
I20260926 13:18:36.182634 20717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a9464d450e447f78bfa49a9b26bb45a"
format_stamp: "Formatted at 2026-09-26 13:18:36 on dist-test-slave-f864"
server_key: "06846c421cb03b3b3dde1aeeb04dd658"
server_key_iv: "af66834cd5c15211c65e1e24c068d185"
server_key_version: "encryptionkey@0"
I20260926 13:18:36.183909 20717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:36.184584 20732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:36.184759 20717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:36.184863 20717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "7a9464d450e447f78bfa49a9b26bb45a"
format_stamp: "Formatted at 2026-09-26 13:18:36 on dist-test-slave-f864"
server_key: "06846c421cb03b3b3dde1aeeb04dd658"
server_key_iv: "af66834cd5c15211c65e1e24c068d185"
server_key_version: "encryptionkey@0"
I20260926 13:18:36.184937 20717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:36.199854 20717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:36.201665 20717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:36.201838 20717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:36.205832 20717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44101
I20260926 13:18:36.205868 20784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44101 every 8 connection(s)
I20260926 13:18:36.206284 20717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:36.207154 20785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:36.209383 20785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a: Bootstrap starting.
I20260926 13:18:36.209944 20785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:36.210243 20785 log.cc:824] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:36.210927 20785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a: No bootstrap required, opened a new log
I20260926 13:18:36.212484 20785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9464d450e447f78bfa49a9b26bb45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44101 } }
I20260926 13:18:36.212617 20785 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:36.212639 20785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a9464d450e447f78bfa49a9b26bb45a, State: Initialized, Role: FOLLOWER
I20260926 13:18:36.212739 20785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9464d450e447f78bfa49a9b26bb45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44101 } }
I20260926 13:18:36.212822 20785 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:36.212859 20785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:36.212929 20785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:36.213243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20717
I20260926 13:18:36.213359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:36.213555 392 external_mini_cluster.cc:1428] Setting key 2cae4668369a111117f430c49a67fc72
I20260926 13:18:36.213685 20785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9464d450e447f78bfa49a9b26bb45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44101 } }
I20260926 13:18:36.213815 20785 leader_election.cc:302] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a9464d450e447f78bfa49a9b26bb45a; no voters:
I20260926 13:18:36.213969 20785 leader_election.cc:288] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:36.214012 20788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:36.214248 20788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [term 1 LEADER]: Becoming Leader. State: Replica: 7a9464d450e447f78bfa49a9b26bb45a, State: Running, Role: LEADER
I20260926 13:18:36.214300 20785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:36.214380 20788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9464d450e447f78bfa49a9b26bb45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44101 } }
I20260926 13:18:36.215152 20790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a9464d450e447f78bfa49a9b26bb45a. Latest consensus state: current_term: 1 leader_uuid: "7a9464d450e447f78bfa49a9b26bb45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9464d450e447f78bfa49a9b26bb45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44101 } } }
I20260926 13:18:36.215283 20790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:36.215129 20792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a9464d450e447f78bfa49a9b26bb45a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a9464d450e447f78bfa49a9b26bb45a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44101 } } }
I20260926 13:18:36.215513 20799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:36.215538 20792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:36.216138 20799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:36.217952 20799 catalog_manager.cc:1347] Generated new cluster ID: 2ec617a6498e4f77bf4684520ac16891
I20260926 13:18:36.218003 20799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:36.219916 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:36.214065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45095 (local address 127.0.0.1:44101)
0926 13:18:36.214249 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:36.214259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:36.214550 (+ 291us) server_negotiation.cc:407] Connection header received
0926 13:18:36.214755 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:36.214765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:36.214839 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:36.214982 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:36.215281 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:36.216698 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:36.217689 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:36.218322 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:36.218442 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:36.219195 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:36.219200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:36.219200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:36.219436 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:36.219475 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:36.219490 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:36.219577 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:36.219669 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:18:36.219699 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260926 13:18:36.220884 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:36.229712 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:36.222456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43931 (local address 127.0.0.1:44101)
0926 13:18:36.222492 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:36.222494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:36.226352 (+ 3858us) server_negotiation.cc:407] Connection header received
0926 13:18:36.226415 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:36.226416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:36.226426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:36.226483 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:36.226612 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:36.227342 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:36.228382 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:36.228703 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:36.228759 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:36.229454 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:36.229455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:36.229455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:36.229459 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:36.229473 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:36.229477 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:36.229533 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:36.229625 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 13:18:36.229638 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:18:36.239790 20799 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:36.240514 20799 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:36.247192 20799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a9464d450e447f78bfa49a9b26bb45a: Generated new TSK 0
I20260926 13:18:36.247397 20799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:36.249287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20717
W20260926 13:18:36.253508 20702 connection.cc:535] client connection to 127.0.0.1:44101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44101): Connection reset by peer (error 104)
2026-09-26T13:18:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:36 dist-test-slave-f864 krb5kdc[20816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:36 dist-test-slave-f864 krb5kdc[20816](info): set up 2 sockets
Sep 26 13:18:36 dist-test-slave-f864 krb5kdc[20816](info): commencing operation
krb5kdc: starting...
W20260926 13:18:37.278298 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:37 dist-test-slave-f864 krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:37.391192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44981
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:37.470777 20829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:37.470988 20829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:37.471011 20829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:37.472172 20829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:37.472221 20829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:37.472236 20829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:37.472278 20829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:37.472306 20829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:37.473721 20829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44981
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:37.473970 20829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:37.474232 20829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:37.478976 20834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:37.478971 20837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:37.479024 20835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:37.479221 20829 server_base.cc:1034] running on GCE node
I20260926 13:18:37.479494 20829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:37.479801 20829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:37.480954 20829 hybrid_clock.cc:648] HybridClock initialized: now 1790428717480935 us; error 33 us; skew 500 ppm
Sep 26 13:18:37 dist-test-slave-f864 krb5kdc[20816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428717, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:37.483394 20829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:37.483806 20829 webserver.cc:460] Webserver started at http://127.0.0.1:44037/ using document root <none> and password file <none>
I20260926 13:18:37.484025 20829 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:37.484093 20829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:37.484241 20829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:37.485258 20829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "1a81caeb9f244372a650a36addd61b1b"
format_stamp: "Formatted at 2026-09-26 13:18:37 on dist-test-slave-f864"
server_key: "f407745a306ebaf0198068f8366d568e"
server_key_iv: "299d4e577ce00916b1cc133a4880ce7a"
server_key_version: "encryptionkey@0"
I20260926 13:18:37.485622 20829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a81caeb9f244372a650a36addd61b1b"
format_stamp: "Formatted at 2026-09-26 13:18:37 on dist-test-slave-f864"
server_key: "f407745a306ebaf0198068f8366d568e"
server_key_iv: "299d4e577ce00916b1cc133a4880ce7a"
server_key_version: "encryptionkey@0"
I20260926 13:18:37.486994 20829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:37.487802 20844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:37.487968 20829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 13:18:37.488065 20829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "1a81caeb9f244372a650a36addd61b1b"
format_stamp: "Formatted at 2026-09-26 13:18:37 on dist-test-slave-f864"
server_key: "f407745a306ebaf0198068f8366d568e"
server_key_iv: "299d4e577ce00916b1cc133a4880ce7a"
server_key_version: "encryptionkey@0"
I20260926 13:18:37.488145 20829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:37.491389 20829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:37.493573 20829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:37.493716 20829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:37.497377 20829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44981
I20260926 13:18:37.497408 20896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44981 every 8 connection(s)
I20260926 13:18:37.497704 20829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:37.498787 20897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:37.500851 20897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b: Bootstrap starting.
I20260926 13:18:37.501374 20897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:37.501662 20897 log.cc:824] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:37.502413 20897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b: No bootstrap required, opened a new log
I20260926 13:18:37.503962 20897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a81caeb9f244372a650a36addd61b1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260926 13:18:37.504088 20897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:37.504109 20897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a81caeb9f244372a650a36addd61b1b, State: Initialized, Role: FOLLOWER
I20260926 13:18:37.504251 20897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a81caeb9f244372a650a36addd61b1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260926 13:18:37.504311 20897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:37.504331 20897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:37.504396 20897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:37.504925 20897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a81caeb9f244372a650a36addd61b1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260926 13:18:37.505031 20897 leader_election.cc:302] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a81caeb9f244372a650a36addd61b1b; no voters:
I20260926 13:18:37.505182 20897 leader_election.cc:288] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:37.505259 20900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:37.505452 20897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:37.505641 20900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [term 1 LEADER]: Becoming Leader. State: Replica: 1a81caeb9f244372a650a36addd61b1b, State: Running, Role: LEADER
I20260926 13:18:37.505681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20829
I20260926 13:18:37.505725 20900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a81caeb9f244372a650a36addd61b1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } }
I20260926 13:18:37.505792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:37.505981 392 external_mini_cluster.cc:1428] Setting key de2d5e701a4490da33aa42d21c477ca4
I20260926 13:18:37.506072 20901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a81caeb9f244372a650a36addd61b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a81caeb9f244372a650a36addd61b1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } } }
I20260926 13:18:37.506148 20901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:37.506335 20902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a81caeb9f244372a650a36addd61b1b. Latest consensus state: current_term: 1 leader_uuid: "1a81caeb9f244372a650a36addd61b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a81caeb9f244372a650a36addd61b1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44981 } } }
I20260926 13:18:37.506395 20902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:37.506825 20909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:37.507459 20909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:37.508491 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:37.506806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52111 (local address 127.0.0.1:44981)
0926 13:18:37.507002 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:37.507012 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:37.507046 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:18:37.507161 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:37.507169 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:37.507202 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:37.507385 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:37.508034 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:37.508037 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:37.508037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:37.508041 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:37.508102 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:37.508118 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:37.508182 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:37.508279 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 13:18:37.508318 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260926 13:18:37.509135 20909 catalog_manager.cc:1347] Generated new cluster ID: f056686950e44fb0a4d8cdc187d49141
I20260926 13:18:37.509184 20909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:37.509446 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:37.511973 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:37.510987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34513 (local address 127.0.0.1:44981)
0926 13:18:37.511016 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:37.511017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:37.511116 (+ 99us) server_negotiation.cc:407] Connection header received
0926 13:18:37.511187 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:37.511187 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:37.511191 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:37.511238 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:37.511782 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:37.511783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:37.511783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:37.511787 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:37.511798 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:37.511802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:37.511851 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:37.511907 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 13:18:37.511919 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 13:18:37.519851 20909 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:37.520011 20909 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:37.525678 20909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a81caeb9f244372a650a36addd61b1b: Generated new TSK 0
I20260926 13:18:37.525833 20909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:37.535791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20829
2026-09-26T13:18:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:37 dist-test-slave-f864 krb5kdc[20928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:37 dist-test-slave-f864 krb5kdc[20928](info): set up 2 sockets
Sep 26 13:18:37 dist-test-slave-f864 krb5kdc[20928](info): commencing operation
krb5kdc: starting...
W20260926 13:18:38.566325 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:38 dist-test-slave-f864 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:38.684553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:38.763056 20941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:38.763304 20941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:38.763340 20941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:38.764472 20941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:38.764546 20941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:38.764569 20941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:38.764591 20941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:38.764619 20941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:38.766063 20941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:38.766369 20941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:38.766592 20941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:38.771440 20946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:38.771441 20949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:38.771508 20947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:38.771497 20941 server_base.cc:1034] running on GCE node
I20260926 13:18:38.771874 20941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:38.772171 20941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:38.773344 20941 hybrid_clock.cc:648] HybridClock initialized: now 1790428718773334 us; error 36 us; skew 500 ppm
Sep 26 13:18:38 dist-test-slave-f864 krb5kdc[20928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428718, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:38.775712 20941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:38.776160 20941 webserver.cc:460] Webserver started at http://127.0.0.1:39997/ using document root <none> and password file <none>
I20260926 13:18:38.776371 20941 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:38.776410 20941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:38.776506 20941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:38.777490 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "8419b621e28f4e1697225b3abc71d0c2"
format_stamp: "Formatted at 2026-09-26 13:18:38 on dist-test-slave-f864"
server_key: "defa1b7da0f35400b0b6243a2865374d"
server_key_iv: "f3244156b7b17d35d32d94d0066a174d"
server_key_version: "encryptionkey@0"
I20260926 13:18:38.777777 20941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "8419b621e28f4e1697225b3abc71d0c2"
format_stamp: "Formatted at 2026-09-26 13:18:38 on dist-test-slave-f864"
server_key: "defa1b7da0f35400b0b6243a2865374d"
server_key_iv: "f3244156b7b17d35d32d94d0066a174d"
server_key_version: "encryptionkey@0"
I20260926 13:18:38.779042 20941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:38.779734 20956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:38.779913 20941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:38.780004 20941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "8419b621e28f4e1697225b3abc71d0c2"
format_stamp: "Formatted at 2026-09-26 13:18:38 on dist-test-slave-f864"
server_key: "defa1b7da0f35400b0b6243a2865374d"
server_key_iv: "f3244156b7b17d35d32d94d0066a174d"
server_key_version: "encryptionkey@0"
I20260926 13:18:38.780076 20941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:38.795262 20941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:38.797590 20941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:38.797748 20941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:38.801628 20941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33351
I20260926 13:18:38.801707 21008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33351 every 8 connection(s)
I20260926 13:18:38.802040 20941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:38.802917 21009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:38.805133 21009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2: Bootstrap starting.
I20260926 13:18:38.805673 21009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:38.805908 21009 log.cc:824] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:38.806566 21009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2: No bootstrap required, opened a new log
I20260926 13:18:38.808116 21009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8419b621e28f4e1697225b3abc71d0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260926 13:18:38.808270 21009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:38.808293 21009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8419b621e28f4e1697225b3abc71d0c2, State: Initialized, Role: FOLLOWER
I20260926 13:18:38.808395 21009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8419b621e28f4e1697225b3abc71d0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260926 13:18:38.808478 21009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:38.808516 21009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:38.808565 21009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:38.808795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 20941
I20260926 13:18:38.808907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:38.809104 392 external_mini_cluster.cc:1428] Setting key f4d031578ad97e2a9a9c0e10024f1d67
I20260926 13:18:38.809211 21009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8419b621e28f4e1697225b3abc71d0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260926 13:18:38.809334 21009 leader_election.cc:302] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8419b621e28f4e1697225b3abc71d0c2; no voters:
I20260926 13:18:38.809500 21009 leader_election.cc:288] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:38.809551 21012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:38.809970 21012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 8419b621e28f4e1697225b3abc71d0c2, State: Running, Role: LEADER
I20260926 13:18:38.810043 21009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:38.810138 21012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8419b621e28f4e1697225b3abc71d0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } }
I20260926 13:18:38.810563 21012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8419b621e28f4e1697225b3abc71d0c2. Latest consensus state: current_term: 1 leader_uuid: "8419b621e28f4e1697225b3abc71d0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8419b621e28f4e1697225b3abc71d0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } } }
I20260926 13:18:38.810765 21012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:38.810700 21016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8419b621e28f4e1697225b3abc71d0c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8419b621e28f4e1697225b3abc71d0c2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33351 } } }
I20260926 13:18:38.811183 21016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:38.811362 21021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:38.811842 21021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:38.813562 21021 catalog_manager.cc:1347] Generated new cluster ID: 5d21cd46d6b6400b8f440a99b6d58dd3
I20260926 13:18:38.813621 21021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:38 dist-test-slave-f864 krb5kdc[20928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:18:38.820040 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:38.809648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41285 (local address 127.0.0.1:33351)
0926 13:18:38.809822 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:38.809833 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:38.809872 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:18:38.809994 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:38.810002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:38.810058 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:38.811164 (+ 1106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:38.811282 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:38.812745 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:38.813864 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:38.814378 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:38.814505 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:38.816802 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:38.816806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:38.816807 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:38.816819 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:38.818567 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:38.818941 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:38.818944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:38.818944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:38.819040 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:38.819247 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:38.819248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:38.819248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:38.819486 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:38.819653 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:38.819746 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 13:18:38.819887 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260926 13:18:38.820899 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:38.832273 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:38.822240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58471 (local address 127.0.0.1:33351)
0926 13:18:38.822275 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:38.822277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:38.826379 (+ 4102us) server_negotiation.cc:407] Connection header received
0926 13:18:38.826455 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:38.826456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:38.826467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:38.826508 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:38.827024 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:38.827528 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:38.828227 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:38.828453 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:38.828538 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:38.829737 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:38.829738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:38.829739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:38.829746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:38.831157 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:38.831495 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:38.831496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:38.831497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:38.831549 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:38.831764 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:38.831764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:38.831765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:38.831871 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:38.831984 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:38.832096 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 13:18:38.832168 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:18:38.840970 21021 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:38.841747 21021 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:38.849803 21021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8419b621e28f4e1697225b3abc71d0c2: Generated new TSK 0
I20260926 13:18:38.850000 21021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:38.858600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 20941
2026-09-26T13:18:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:38 dist-test-slave-f864 krb5kdc[21040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:38 dist-test-slave-f864 krb5kdc[21040](info): set up 2 sockets
Sep 26 13:18:38 dist-test-slave-f864 krb5kdc[21040](info): commencing operation
krb5kdc: starting...
W20260926 13:18:39.914325 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:39 dist-test-slave-f864 krb5kdc[21040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428719, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:40.034139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:40.116916 21053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:40.117152 21053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:40.117190 21053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:40.118396 21053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:40.118456 21053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:40.118481 21053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:40.118510 21053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:40.118541 21053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:40.119994 21053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:40.120257 21053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:40.120481 21053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:40.125149 21061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:40.125166 21059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:40.125349 21053 server_base.cc:1034] running on GCE node
W20260926 13:18:40.125157 21058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:40.125618 21053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:40.125895 21053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:40.127071 21053 hybrid_clock.cc:648] HybridClock initialized: now 1790428720127062 us; error 33 us; skew 500 ppm
Sep 26 13:18:40 dist-test-slave-f864 krb5kdc[21040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428720, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:40.136361 21053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:40.136813 21053 webserver.cc:460] Webserver started at http://127.0.0.1:41531/ using document root <none> and password file <none>
I20260926 13:18:40.137059 21053 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:40.137128 21053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:40.137274 21053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:40.138438 21053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "0de9998dee69462482fff4468da12a71"
format_stamp: "Formatted at 2026-09-26 13:18:40 on dist-test-slave-f864"
server_key: "c637ac64876940f181357e5f8426f607"
server_key_iv: "a21f021100534b63f993cda3ad4b3fb8"
server_key_version: "encryptionkey@0"
I20260926 13:18:40.138805 21053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "0de9998dee69462482fff4468da12a71"
format_stamp: "Formatted at 2026-09-26 13:18:40 on dist-test-slave-f864"
server_key: "c637ac64876940f181357e5f8426f607"
server_key_iv: "a21f021100534b63f993cda3ad4b3fb8"
server_key_version: "encryptionkey@0"
I20260926 13:18:40.140108 21053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:40.140844 21068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:40.141011 21053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:40.141117 21053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "0de9998dee69462482fff4468da12a71"
format_stamp: "Formatted at 2026-09-26 13:18:40 on dist-test-slave-f864"
server_key: "c637ac64876940f181357e5f8426f607"
server_key_iv: "a21f021100534b63f993cda3ad4b3fb8"
server_key_version: "encryptionkey@0"
I20260926 13:18:40.141191 21053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:40.160660 21053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:40.162984 21053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:40.163159 21053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:40.167142 21053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33453
I20260926 13:18:40.167214 21120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33453 every 8 connection(s)
I20260926 13:18:40.167538 21053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:40.168355 21121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:40.168722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21053
I20260926 13:18:40.168843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:40.169030 392 external_mini_cluster.cc:1428] Setting key ec1d864ead436adbab1f5475ae0cdc2d
I20260926 13:18:40.171214 21121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71: Bootstrap starting.
I20260926 13:18:40.171892 21121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:40.172194 21121 log.cc:824] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:40.172887 21121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71: No bootstrap required, opened a new log
I20260926 13:18:40.174546 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:40.169572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37165 (local address 127.0.0.1:33453)
0926 13:18:40.169733 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:40.169742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:40.169775 (+ 33us) server_negotiation.cc:407] Connection header received
0926 13:18:40.170214 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:40.170223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:40.170282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:40.170396 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:40.170653 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:40.171569 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:40.172672 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:40.173092 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:40.173175 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:40.173801 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:40.173805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:40.173805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:40.173997 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:40.174038 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:40.174049 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:40.174104 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:40.174275 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 13:18:40.174322 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260926 13:18:40.175410 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:40.175307 21121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de9998dee69462482fff4468da12a71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 13:18:40.175467 21121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:40.175498 21121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0de9998dee69462482fff4468da12a71, State: Initialized, Role: FOLLOWER
I20260926 13:18:40.175611 21121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de9998dee69462482fff4468da12a71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 13:18:40.175678 21121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:40.175704 21121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:40.175745 21121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:40.176414 21121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de9998dee69462482fff4468da12a71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 13:18:40.176537 21121 leader_election.cc:302] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0de9998dee69462482fff4468da12a71; no voters:
I20260926 13:18:40.176716 21121 leader_election.cc:288] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:40.177035 21121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:40.176753 21130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:40.178412 21130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [term 1 LEADER]: Becoming Leader. State: Replica: 0de9998dee69462482fff4468da12a71, State: Running, Role: LEADER
I20260926 13:18:40.178504 21130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de9998dee69462482fff4468da12a71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } }
I20260926 13:18:40.179014 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0de9998dee69462482fff4468da12a71. Latest consensus state: current_term: 1 leader_uuid: "0de9998dee69462482fff4468da12a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de9998dee69462482fff4468da12a71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } } }
I20260926 13:18:40.179103 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:40.179061 21131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0de9998dee69462482fff4468da12a71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0de9998dee69462482fff4468da12a71" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33453 } } }
I20260926 13:18:40.179294 21131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71 [sys.catalog]: This master's current role is: LEADER
W20260926 13:18:40.179705 21145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:18:40.179767 21145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:18:40.179857 21146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:40.180130 21146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:40.180325 21124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:40.176865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54271 (local address 127.0.0.1:33453)
0926 13:18:40.176895 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:40.176897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:40.177127 (+ 230us) server_negotiation.cc:407] Connection header received
0926 13:18:40.177152 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:40.177153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:40.177163 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:40.177240 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:40.177357 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:40.178114 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:40.178978 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:40.179278 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:40.179341 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:40.180077 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:40.180079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:40.180079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:40.180083 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:40.180095 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:40.180100 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:40.180159 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:40.180233 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 13:18:40.180253 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":14}
I20260926 13:18:40.181572 21146 catalog_manager.cc:1347] Generated new cluster ID: a9c4640e03c94fad810c1cc5d839ef09
I20260926 13:18:40.181627 21146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:40.195662 21146 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:40.196228 21146 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:40.205924 21146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0de9998dee69462482fff4468da12a71: Generated new TSK 0
I20260926 13:18:40.206148 21146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:40.215916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21053
2026-09-26T13:18:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:40 dist-test-slave-f864 krb5kdc[21152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:40 dist-test-slave-f864 krb5kdc[21152](info): set up 2 sockets
Sep 26 13:18:40 dist-test-slave-f864 krb5kdc[21152](info): commencing operation
krb5kdc: starting...
W20260926 13:18:41.254504 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:41 dist-test-slave-f864 krb5kdc[21152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428721, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:41.373148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36337
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:41.452479 21165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:41.452723 21165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:41.452762 21165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:41.453883 21165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:41.453951 21165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:41.453976 21165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:41.454005 21165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:41.454035 21165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:41.455484 21165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:41.455762 21165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:41.455974 21165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:41.460860 21170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:41.460903 21171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:41.460916 21165 server_base.cc:1034] running on GCE node
W20260926 13:18:41.460877 21173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:41.461305 21165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:41.461615 21165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:41.462786 21165 hybrid_clock.cc:648] HybridClock initialized: now 1790428721462763 us; error 35 us; skew 500 ppm
Sep 26 13:18:41 dist-test-slave-f864 krb5kdc[21152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428721, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:41.465145 21165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:41.465513 21165 webserver.cc:460] Webserver started at http://127.0.0.1:37775/ using document root <none> and password file <none>
I20260926 13:18:41.465698 21165 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:41.465736 21165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:41.465822 21165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:41.466874 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "95562320f655454ab14e63e1ceff1bfa"
format_stamp: "Formatted at 2026-09-26 13:18:41 on dist-test-slave-f864"
server_key: "18cabe2339d2b2ba7bccd130892a0ed3"
server_key_iv: "8a9a758bb79e5836323bdbb885d2e803"
server_key_version: "encryptionkey@0"
I20260926 13:18:41.467212 21165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "95562320f655454ab14e63e1ceff1bfa"
format_stamp: "Formatted at 2026-09-26 13:18:41 on dist-test-slave-f864"
server_key: "18cabe2339d2b2ba7bccd130892a0ed3"
server_key_iv: "8a9a758bb79e5836323bdbb885d2e803"
server_key_version: "encryptionkey@0"
I20260926 13:18:41.468426 21165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:41.469108 21180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:41.469296 21165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:41.469380 21165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "95562320f655454ab14e63e1ceff1bfa"
format_stamp: "Formatted at 2026-09-26 13:18:41 on dist-test-slave-f864"
server_key: "18cabe2339d2b2ba7bccd130892a0ed3"
server_key_iv: "8a9a758bb79e5836323bdbb885d2e803"
server_key_version: "encryptionkey@0"
I20260926 13:18:41.469434 21165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:41.472626 21165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:41.474495 21165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:41.474602 21165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:41.478677 21165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36337
I20260926 13:18:41.478729 21232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36337 every 8 connection(s)
I20260926 13:18:41.479036 21165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:41.480000 21233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:41.482216 21233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa: Bootstrap starting.
I20260926 13:18:41.482805 21233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:41.483081 21233 log.cc:824] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:41.483749 21233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa: No bootstrap required, opened a new log
I20260926 13:18:41.485327 21233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95562320f655454ab14e63e1ceff1bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260926 13:18:41.485461 21233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:41.485483 21233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95562320f655454ab14e63e1ceff1bfa, State: Initialized, Role: FOLLOWER
I20260926 13:18:41.485638 21233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95562320f655454ab14e63e1ceff1bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260926 13:18:41.485702 21233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:41.485723 21233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:41.485792 21233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:41.486428 21233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95562320f655454ab14e63e1ceff1bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260926 13:18:41.486538 21233 leader_election.cc:302] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95562320f655454ab14e63e1ceff1bfa; no voters:
I20260926 13:18:41.486749 21233 leader_election.cc:288] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:41.486794 21236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:41.486974 21236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [term 1 LEADER]: Becoming Leader. State: Replica: 95562320f655454ab14e63e1ceff1bfa, State: Running, Role: LEADER
I20260926 13:18:41.487007 21233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:41.487145 21236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95562320f655454ab14e63e1ceff1bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } }
I20260926 13:18:41.487310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21165
I20260926 13:18:41.487426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:41.487612 392 external_mini_cluster.cc:1428] Setting key 32e0940913f8989051e6fb1aa30024f9
I20260926 13:18:41.487840 21238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95562320f655454ab14e63e1ceff1bfa. Latest consensus state: current_term: 1 leader_uuid: "95562320f655454ab14e63e1ceff1bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95562320f655454ab14e63e1ceff1bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } } }
I20260926 13:18:41.487953 21238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:41.488037 21237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95562320f655454ab14e63e1ceff1bfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95562320f655454ab14e63e1ceff1bfa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36337 } } }
I20260926 13:18:41.488131 21237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:41.488636 21246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:41.489182 21246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:41.490316 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:41.488286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42005 (local address 127.0.0.1:36337)
0926 13:18:41.488681 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:41.488692 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:41.488729 (+ 37us) server_negotiation.cc:407] Connection header received
0926 13:18:41.488857 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:41.488864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:41.488897 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:41.489085 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:41.489840 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:41.489844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:41.489844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:41.489849 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:41.489898 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:41.489910 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:41.489943 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:41.490072 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 13:18:41.490119 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":79,"threads_started":1}
I20260926 13:18:41.490942 21246 catalog_manager.cc:1347] Generated new cluster ID: bd2f130796954c258188cc316ce69c63
I20260926 13:18:41.490990 21246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:41.494517 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:41.499758 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:41.498565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38675 (local address 127.0.0.1:36337)
0926 13:18:41.498610 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:41.498612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:41.498750 (+ 138us) server_negotiation.cc:407] Connection header received
0926 13:18:41.498827 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:41.498828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:41.498833 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:41.498889 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:41.499504 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:41.499505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:41.499506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:41.499510 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:41.499535 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:41.499540 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:41.499592 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:41.499661 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 13:18:41.499675 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:18:41.509409 21246 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:41.509606 21246 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:41.519606 21246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95562320f655454ab14e63e1ceff1bfa: Generated new TSK 0
I20260926 13:18:41.519821 21246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:41.527174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21165
2026-09-26T13:18:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:41 dist-test-slave-f864 krb5kdc[21264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:41 dist-test-slave-f864 krb5kdc[21264](info): set up 2 sockets
Sep 26 13:18:41 dist-test-slave-f864 krb5kdc[21264](info): commencing operation
krb5kdc: starting...
W20260926 13:18:42.562381 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:42 dist-test-slave-f864 krb5kdc[21264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:42.684157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45215
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:42.766364 21277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:42.766672 21277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:42.766721 21277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:42.767997 21277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:42.768074 21277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:42.768097 21277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:42.768119 21277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:42.768148 21277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:42.769747 21277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:42.770040 21277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:42.770305 21277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:18:42.775374 21277 server_base.cc:1034] running on GCE node
W20260926 13:18:42.775314 21282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:42.775341 21283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:42.775314 21285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:42.775758 21277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:42.776089 21277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:42.777279 21277 hybrid_clock.cc:648] HybridClock initialized: now 1790428722777255 us; error 36 us; skew 500 ppm
Sep 26 13:18:42 dist-test-slave-f864 krb5kdc[21264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:42.780067 21277 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:42.780508 21277 webserver.cc:460] Webserver started at http://127.0.0.1:41495/ using document root <none> and password file <none>
I20260926 13:18:42.780748 21277 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:42.780821 21277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:42.780980 21277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:42.782140 21277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "1cd65a3b68fd4a2dae875121d5cdd630"
format_stamp: "Formatted at 2026-09-26 13:18:42 on dist-test-slave-f864"
server_key: "ce626af30e7419c2cb2a40be75269725"
server_key_iv: "9fd5c430321a0c606c862c093342cc49"
server_key_version: "encryptionkey@0"
I20260926 13:18:42.782642 21277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "1cd65a3b68fd4a2dae875121d5cdd630"
format_stamp: "Formatted at 2026-09-26 13:18:42 on dist-test-slave-f864"
server_key: "ce626af30e7419c2cb2a40be75269725"
server_key_iv: "9fd5c430321a0c606c862c093342cc49"
server_key_version: "encryptionkey@0"
I20260926 13:18:42.784016 21277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 13:18:42.784807 21292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:42.785000 21277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:42.785109 21277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "1cd65a3b68fd4a2dae875121d5cdd630"
format_stamp: "Formatted at 2026-09-26 13:18:42 on dist-test-slave-f864"
server_key: "ce626af30e7419c2cb2a40be75269725"
server_key_iv: "9fd5c430321a0c606c862c093342cc49"
server_key_version: "encryptionkey@0"
I20260926 13:18:42.785198 21277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:42.794968 21277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:42.797364 21277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:42.797540 21277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:42.801519 21277 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45215
I20260926 13:18:42.801560 21344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45215 every 8 connection(s)
I20260926 13:18:42.801944 21277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:42.802842 21345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:42.805387 21345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630: Bootstrap starting.
I20260926 13:18:42.806128 21345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:42.806478 21345 log.cc:824] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:42.807181 21345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630: No bootstrap required, opened a new log
I20260926 13:18:42.808764 21345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260926 13:18:42.808914 21345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:42.808946 21345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd65a3b68fd4a2dae875121d5cdd630, State: Initialized, Role: FOLLOWER
I20260926 13:18:42.808979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21277
I20260926 13:18:42.809116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:42.809123 21345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260926 13:18:42.809214 21345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:42.809280 21345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:42.809300 392 external_mini_cluster.cc:1428] Setting key e44840d9245e33e8e1006a945f0cbd0f
I20260926 13:18:42.809341 21345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:42.810647 21345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260926 13:18:42.810779 21345 leader_election.cc:302] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cd65a3b68fd4a2dae875121d5cdd630; no voters:
I20260926 13:18:42.810951 21345 leader_election.cc:288] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:42.810994 21350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:42.811198 21350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [term 1 LEADER]: Becoming Leader. State: Replica: 1cd65a3b68fd4a2dae875121d5cdd630, State: Running, Role: LEADER
I20260926 13:18:42.811260 21345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:42.811331 21350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } }
I20260926 13:18:42.811825 21351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } } }
I20260926 13:18:42.811815 21350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cd65a3b68fd4a2dae875121d5cdd630. Latest consensus state: current_term: 1 leader_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cd65a3b68fd4a2dae875121d5cdd630" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45215 } } }
I20260926 13:18:42.811930 21351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:42.811939 21350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:42.812199 21359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:42.812764 21359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:42.814694 21359 catalog_manager.cc:1347] Generated new cluster ID: 5d79c95d923e44cb9969d43c330ba95a
I20260926 13:18:42.814754 21359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:42 dist-test-slave-f864 krb5kdc[21264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:18:42.822254 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:42.810045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:51381 (local address 127.0.0.1:45215)
0926 13:18:42.810286 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:42.810296 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:42.810336 (+ 40us) server_negotiation.cc:407] Connection header received
0926 13:18:42.810442 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:42.810448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:42.810494 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:42.810612 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:42.811513 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:42.813116 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:42.814152 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:42.814811 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:42.814949 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:42.817971 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:42.817975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:42.817976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:42.817990 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:42.820296 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:42.820795 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:42.820808 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:42.820808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:42.820957 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:42.821219 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:42.821220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:42.821221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:42.821570 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:42.821740 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:42.821868 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 13:18:42.821984 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":151,"threads_started":1}
I20260926 13:18:42.823410 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:42.828786 21359 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:42.829353 21359 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:42.832963 21349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:42.824965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:48293 (local address 127.0.0.1:45215)
0926 13:18:42.825007 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:42.825009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:42.825123 (+ 114us) server_negotiation.cc:407] Connection header received
0926 13:18:42.825185 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:42.825186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:42.825215 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:42.825287 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:42.825941 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:42.826723 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:42.827833 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:42.828148 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:42.828238 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:42.829947 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:42.829948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:42.829949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:42.829956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:42.831632 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:42.832095 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:42.832096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:42.832096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:42.832165 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:42.832409 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:42.832409 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:42.832410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:42.832540 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:42.832646 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:42.832806 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 13:18:42.832868 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:18:42.837049 21359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cd65a3b68fd4a2dae875121d5cdd630: Generated new TSK 0
I20260926 13:18:42.837285 21359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:42.842325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21277
W20260926 13:18:42.846364 21262 connection.cc:535] client connection to 127.0.0.1:45215 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45215): Connection reset by peer (error 104)
2026-09-26T13:18:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:42 dist-test-slave-f864 krb5kdc[21376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:42 dist-test-slave-f864 krb5kdc[21376](info): set up 2 sockets
Sep 26 13:18:42 dist-test-slave-f864 krb5kdc[21376](info): commencing operation
krb5kdc: starting...
W20260926 13:18:43.890264 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:43 dist-test-slave-f864 krb5kdc[21376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428723, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:44.008774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39925
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:44.093148 21389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:44.093379 21389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:44.093402 21389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:44.094630 21389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:44.094672 21389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:44.094686 21389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:44.094700 21389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:44.094749 21389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:44.096180 21389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:44.096428 21389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:44.096655 21389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:44.101662 21394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:44.101645 21397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:44.101645 21395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:44.101673 21389 server_base.cc:1034] running on GCE node
I20260926 13:18:44.102052 21389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:44.102401 21389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:44.103555 21389 hybrid_clock.cc:648] HybridClock initialized: now 1790428724103539 us; error 34 us; skew 500 ppm
Sep 26 13:18:44 dist-test-slave-f864 krb5kdc[21376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428724, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:44.105978 21389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:44.106459 21389 webserver.cc:460] Webserver started at http://127.0.0.1:34385/ using document root <none> and password file <none>
I20260926 13:18:44.106691 21389 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:44.106758 21389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:44.106912 21389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:44.107997 21389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "27032869cdb34e088b9ab5e6f406f287"
format_stamp: "Formatted at 2026-09-26 13:18:44 on dist-test-slave-f864"
server_key: "af11d742cbc58c09fbe966954dc119b7"
server_key_iv: "9119ec69d0f1aa076e5e601024f32419"
server_key_version: "encryptionkey@0"
I20260926 13:18:44.108361 21389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "27032869cdb34e088b9ab5e6f406f287"
format_stamp: "Formatted at 2026-09-26 13:18:44 on dist-test-slave-f864"
server_key: "af11d742cbc58c09fbe966954dc119b7"
server_key_iv: "9119ec69d0f1aa076e5e601024f32419"
server_key_version: "encryptionkey@0"
I20260926 13:18:44.109654 21389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:44.110406 21404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:44.110605 21389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:44.110710 21389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "27032869cdb34e088b9ab5e6f406f287"
format_stamp: "Formatted at 2026-09-26 13:18:44 on dist-test-slave-f864"
server_key: "af11d742cbc58c09fbe966954dc119b7"
server_key_iv: "9119ec69d0f1aa076e5e601024f32419"
server_key_version: "encryptionkey@0"
I20260926 13:18:44.110783 21389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:44.124682 21389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:44.126950 21389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:44.127127 21389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:44.131135 21389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39925
I20260926 13:18:44.131173 21456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39925 every 8 connection(s)
I20260926 13:18:44.131577 21389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:44.132431 21457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:44.133689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21389
I20260926 13:18:44.133806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:44.133987 392 external_mini_cluster.cc:1428] Setting key 853bfd68e1efa623d1c34cbf67eb339d
I20260926 13:18:44.135193 21457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287: Bootstrap starting.
I20260926 13:18:44.135927 21457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:44.136238 21457 log.cc:824] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:44.136999 21457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287: No bootstrap required, opened a new log
I20260926 13:18:44.139189 21457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27032869cdb34e088b9ab5e6f406f287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39925 } }
I20260926 13:18:44.139380 21457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:44.139426 21457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27032869cdb34e088b9ab5e6f406f287, State: Initialized, Role: FOLLOWER
I20260926 13:18:44.139592 21457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27032869cdb34e088b9ab5e6f406f287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39925 } }
I20260926 13:18:44.139688 21457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:44.139734 21457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:44.139781 21457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:44.140098 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:44.134554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:59065 (local address 127.0.0.1:39925)
0926 13:18:44.134764 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:44.134776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:44.134812 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:18:44.135365 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:44.135374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:44.135436 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:44.135606 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:44.135754 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:44.137099 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:44.137860 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:44.138529 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:44.138667 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:44.139322 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:44.139326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:44.139326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:44.139542 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:44.139582 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:44.139597 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:44.139680 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:44.139834 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 13:18:44.139866 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260926 13:18:44.140619 21457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27032869cdb34e088b9ab5e6f406f287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39925 } }
I20260926 13:18:44.140725 21457 leader_election.cc:302] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27032869cdb34e088b9ab5e6f406f287; no voters:
I20260926 13:18:44.140925 21457 leader_election.cc:288] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:44.140945 21462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:44.141067 21462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [term 1 LEADER]: Becoming Leader. State: Replica: 27032869cdb34e088b9ab5e6f406f287, State: Running, Role: LEADER
I20260926 13:18:44.141203 21457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:44.141189 21462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27032869cdb34e088b9ab5e6f406f287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39925 } }
I20260926 13:18:44.141340 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:44.141510 21464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27032869cdb34e088b9ab5e6f406f287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27032869cdb34e088b9ab5e6f406f287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39925 } } }
I20260926 13:18:44.141716 21464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:44.141657 21463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27032869cdb34e088b9ab5e6f406f287. Latest consensus state: current_term: 1 leader_uuid: "27032869cdb34e088b9ab5e6f406f287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27032869cdb34e088b9ab5e6f406f287" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39925 } } }
I20260926 13:18:44.141995 21463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:44.142446 21466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:44.143004 21466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:44.144644 21466 catalog_manager.cc:1347] Generated new cluster ID: c7cca3fd5b7f4c3aa05627000ecb8bc5
I20260926 13:18:44.144693 21466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:44.146776 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:44.143040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:44815 (local address 127.0.0.1:39925)
0926 13:18:44.143094 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:44.143096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:44.143170 (+ 74us) server_negotiation.cc:407] Connection header received
0926 13:18:44.143268 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:44.143269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:44.143280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:44.143309 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:44.143514 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:44.144254 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:44.145290 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:44.145585 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:44.145629 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:44.146499 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:44.146500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:44.146501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:44.146506 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:44.146520 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:44.146524 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:44.146557 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:44.146690 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:18:44.146703 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 13:18:44.153661 21466 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:44.154254 21466 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:44.157184 21466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27032869cdb34e088b9ab5e6f406f287: Generated new TSK 0
I20260926 13:18:44.157368 21466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:44.160451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21389
W20260926 13:18:44.164985 21374 connection.cc:535] client connection to 127.0.0.1:39925 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39925): Connection reset by peer (error 104)
2026-09-26T13:18:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:44 dist-test-slave-f864 krb5kdc[21488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:44 dist-test-slave-f864 krb5kdc[21488](info): set up 2 sockets
Sep 26 13:18:44 dist-test-slave-f864 krb5kdc[21488](info): commencing operation
krb5kdc: starting...
W20260926 13:18:45.226413 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:45 dist-test-slave-f864 krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:45.348677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:45.429497 21501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:45.429720 21501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:45.429745 21501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:45.430864 21501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:45.430902 21501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:45.430917 21501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:45.430935 21501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:45.430948 21501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:45.432340 21501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42041
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:45.432534 21501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:45.432792 21501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:45.437849 21506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:45.437840 21509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:45.437840 21507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:45.437868 21501 server_base.cc:1034] running on GCE node
I20260926 13:18:45.438359 21501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:45.438651 21501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:45.439818 21501 hybrid_clock.cc:648] HybridClock initialized: now 1790428725439805 us; error 44 us; skew 500 ppm
Sep 26 13:18:45 dist-test-slave-f864 krb5kdc[21488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428725, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:45.447541 21501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:45.448000 21501 webserver.cc:460] Webserver started at http://127.0.0.1:46305/ using document root <none> and password file <none>
I20260926 13:18:45.448251 21501 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:45.448319 21501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:45.448474 21501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:45.449573 21501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "748041650e4d43e99e32a2ac8abb98ae"
format_stamp: "Formatted at 2026-09-26 13:18:45 on dist-test-slave-f864"
server_key: "e55937b0b3622060a800f4e98eac6fd5"
server_key_iv: "121e81c6bcc01c43da2bbd29211f0303"
server_key_version: "encryptionkey@0"
I20260926 13:18:45.449944 21501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "748041650e4d43e99e32a2ac8abb98ae"
format_stamp: "Formatted at 2026-09-26 13:18:45 on dist-test-slave-f864"
server_key: "e55937b0b3622060a800f4e98eac6fd5"
server_key_iv: "121e81c6bcc01c43da2bbd29211f0303"
server_key_version: "encryptionkey@0"
I20260926 13:18:45.451397 21501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:45.452212 21516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:45.452425 21501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:45.452527 21501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "748041650e4d43e99e32a2ac8abb98ae"
format_stamp: "Formatted at 2026-09-26 13:18:45 on dist-test-slave-f864"
server_key: "e55937b0b3622060a800f4e98eac6fd5"
server_key_iv: "121e81c6bcc01c43da2bbd29211f0303"
server_key_version: "encryptionkey@0"
I20260926 13:18:45.452615 21501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:45.456127 21501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:45.458257 21501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:45.458439 21501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:45.462468 21501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42041
I20260926 13:18:45.462504 21568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42041 every 8 connection(s)
I20260926 13:18:45.462880 21501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:45.463152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21501
I20260926 13:18:45.463271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:45.463490 392 external_mini_cluster.cc:1428] Setting key cf731d9a99480a4a822adec3a48645ff
I20260926 13:18:45.464174 21569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:45.466042 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:45.464071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:55105 (local address 127.0.0.1:42041)
0926 13:18:45.464262 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:45.464272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:45.464307 (+ 35us) server_negotiation.cc:407] Connection header received
0926 13:18:45.464747 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:45.464754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:45.464780 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:45.464901 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:45.465579 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:45.465582 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:45.465583 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:45.465586 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:45.465623 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:45.465634 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:45.465689 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:45.465784 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 13:18:45.465851 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260926 13:18:45.467052 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:45.467512 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae: Bootstrap starting.
I20260926 13:18:45.468251 21569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:45.468606 21569 log.cc:824] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:45.469352 21569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae: No bootstrap required, opened a new log
I20260926 13:18:45.469803 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:45.468463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:38015 (local address 127.0.0.1:42041)
0926 13:18:45.468542 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:45.468544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:45.468888 (+ 344us) server_negotiation.cc:407] Connection header received
0926 13:18:45.468912 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:45.468913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:45.468918 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:45.468985 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:45.469610 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:45.469612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:45.469612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:45.469616 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:45.469631 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:45.469635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:45.469701 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:45.469735 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 13:18:45.469748 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:18:45.471562 21569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748041650e4d43e99e32a2ac8abb98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260926 13:18:45.471726 21569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:45.471774 21569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748041650e4d43e99e32a2ac8abb98ae, State: Initialized, Role: FOLLOWER
I20260926 13:18:45.471902 21569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748041650e4d43e99e32a2ac8abb98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260926 13:18:45.471993 21569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:45.472038 21569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:45.472087 21569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:45.472636 21569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748041650e4d43e99e32a2ac8abb98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260926 13:18:45.472756 21569 leader_election.cc:302] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 748041650e4d43e99e32a2ac8abb98ae; no voters:
I20260926 13:18:45.472976 21569 leader_election.cc:288] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:45.473083 21579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:45.473237 21579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [term 1 LEADER]: Becoming Leader. State: Replica: 748041650e4d43e99e32a2ac8abb98ae, State: Running, Role: LEADER
I20260926 13:18:45.473269 21569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:45.473330 21579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748041650e4d43e99e32a2ac8abb98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } }
I20260926 13:18:45.473758 21580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "748041650e4d43e99e32a2ac8abb98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748041650e4d43e99e32a2ac8abb98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } } }
I20260926 13:18:45.473795 21581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748041650e4d43e99e32a2ac8abb98ae. Latest consensus state: current_term: 1 leader_uuid: "748041650e4d43e99e32a2ac8abb98ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748041650e4d43e99e32a2ac8abb98ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42041 } } }
I20260926 13:18:45.473914 21580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:45.473933 21581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:45.474519 21588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:45.475111 21588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:45.476774 21588 catalog_manager.cc:1347] Generated new cluster ID: e8b62660f9af4f16b3a17f6b393a9628
I20260926 13:18:45.476828 21588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:45.499948 21588 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:45.500196 21588 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:45.506055 21588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 748041650e4d43e99e32a2ac8abb98ae: Generated new TSK 0
I20260926 13:18:45.506309 21588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:45.516222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21501
2026-09-26T13:18:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:45 dist-test-slave-f864 krb5kdc[21600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:45 dist-test-slave-f864 krb5kdc[21600](info): set up 2 sockets
Sep 26 13:18:45 dist-test-slave-f864 krb5kdc[21600](info): commencing operation
krb5kdc: starting...
W20260926 13:18:46.550244 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:46 dist-test-slave-f864 krb5kdc[21600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:46.668569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:46.746676 21613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:46.746908 21613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:46.746946 21613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:46.748097 21613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:46.748168 21613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:46.748191 21613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:46.748212 21613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:46.748241 21613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:46.749711 21613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:46.749980 21613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:46.750228 21613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:46.755174 21621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:46.755223 21619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:46.755174 21618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:46.755470 21613 server_base.cc:1034] running on GCE node
I20260926 13:18:46.755707 21613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:46.756018 21613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:46.757206 21613 hybrid_clock.cc:648] HybridClock initialized: now 1790428726757192 us; error 26 us; skew 500 ppm
Sep 26 13:18:46 dist-test-slave-f864 krb5kdc[21600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428726, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:46.759639 21613 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:46.760097 21613 webserver.cc:460] Webserver started at http://127.0.0.1:33791/ using document root <none> and password file <none>
I20260926 13:18:46.760331 21613 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:46.760397 21613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:46.760553 21613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:46.761579 21613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "fbdf14b46c4541f6b3959cf4616abefd"
format_stamp: "Formatted at 2026-09-26 13:18:46 on dist-test-slave-f864"
server_key: "32458cefc18219f0670179905b1581cb"
server_key_iv: "269c9fb3494bdafe7a231fab30775ef5"
server_key_version: "encryptionkey@0"
I20260926 13:18:46.761929 21613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "fbdf14b46c4541f6b3959cf4616abefd"
format_stamp: "Formatted at 2026-09-26 13:18:46 on dist-test-slave-f864"
server_key: "32458cefc18219f0670179905b1581cb"
server_key_iv: "269c9fb3494bdafe7a231fab30775ef5"
server_key_version: "encryptionkey@0"
I20260926 13:18:46.763307 21613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:46.764117 21628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:46.764508 21613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:46.764606 21613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "fbdf14b46c4541f6b3959cf4616abefd"
format_stamp: "Formatted at 2026-09-26 13:18:46 on dist-test-slave-f864"
server_key: "32458cefc18219f0670179905b1581cb"
server_key_iv: "269c9fb3494bdafe7a231fab30775ef5"
server_key_version: "encryptionkey@0"
I20260926 13:18:46.764710 21613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:46.781893 21613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:46.783931 21613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:46.784107 21613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:46.788434 21613 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45025
I20260926 13:18:46.788486 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45025 every 8 connection(s)
I20260926 13:18:46.788870 21613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:46.789673 21681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:46.791870 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd: Bootstrap starting.
I20260926 13:18:46.792428 21681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:46.792707 21681 log.cc:824] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:46.793080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21613
I20260926 13:18:46.793203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:46.793449 392 external_mini_cluster.cc:1428] Setting key 186fa6c5eba833da4d2b53ba713fabe1
I20260926 13:18:46.793572 21681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd: No bootstrap required, opened a new log
I20260926 13:18:46.795683 21681 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdf14b46c4541f6b3959cf4616abefd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260926 13:18:46.795858 21681 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:46.795907 21681 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbdf14b46c4541f6b3959cf4616abefd, State: Initialized, Role: FOLLOWER
I20260926 13:18:46.796047 21681 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdf14b46c4541f6b3959cf4616abefd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260926 13:18:46.796136 21681 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:46.796182 21681 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:46.796228 21681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:46.796877 21681 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdf14b46c4541f6b3959cf4616abefd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260926 13:18:46.796991 21681 leader_election.cc:302] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbdf14b46c4541f6b3959cf4616abefd; no voters:
I20260926 13:18:46.797129 21681 leader_election.cc:288] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:46.797277 21687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:46.797340 21681 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:46.797431 21687 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [term 1 LEADER]: Becoming Leader. State: Replica: fbdf14b46c4541f6b3959cf4616abefd, State: Running, Role: LEADER
I20260926 13:18:46.797523 21687 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdf14b46c4541f6b3959cf4616abefd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } }
I20260926 13:18:46.797874 21688 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbdf14b46c4541f6b3959cf4616abefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdf14b46c4541f6b3959cf4616abefd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } } }
I20260926 13:18:46.797855 21687 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbdf14b46c4541f6b3959cf4616abefd. Latest consensus state: current_term: 1 leader_uuid: "fbdf14b46c4541f6b3959cf4616abefd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbdf14b46c4541f6b3959cf4616abefd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45025 } } }
I20260926 13:18:46.798028 21687 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:46.798239 21688 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:46.798241 21695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:46.798772 21695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:46.800752 21695 catalog_manager.cc:1347] Generated new cluster ID: c9af2dd86a4140c681630bec13708a3c
I20260926 13:18:46.800807 21695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:46 dist-test-slave-f864 krb5kdc[21600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428726, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 13:18:46.806094 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:46.793969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:40377 (local address 127.0.0.1:45025)
0926 13:18:46.794155 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:46.794186 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:46.794225 (+ 39us) server_negotiation.cc:407] Connection header received
0926 13:18:46.794950 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:46.794959 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:46.795022 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:46.795168 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:46.795802 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:46.797534 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:46.798801 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:46.799662 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:46.799776 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:46.802508 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:46.802511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:46.802512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:46.802522 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:46.804440 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:46.804857 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:46.804860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:46.804860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:46.804964 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:46.805215 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:46.805216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:46.805216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:46.805526 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:46.805684 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:46.805802 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 13:18:46.805900 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260926 13:18:46.807334 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:46.811518 21695 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:46.812139 21695 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:46.815486 21685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:46.808748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:59163 (local address 127.0.0.1:45025)
0926 13:18:46.808789 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:46.808790 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:46.808916 (+ 126us) server_negotiation.cc:407] Connection header received
0926 13:18:46.808970 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:46.808970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:46.808979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:46.809029 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:46.809590 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:46.810354 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:46.811311 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:46.811625 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:46.811687 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:46.812836 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:46.812837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:46.812837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:46.812843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:46.814310 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:46.814659 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:46.814660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:46.814660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:46.814711 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:46.814961 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:46.814962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:46.814963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:46.815070 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:46.815191 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:46.815329 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 13:18:46.815388 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:18:46.818257 21695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbdf14b46c4541f6b3959cf4616abefd: Generated new TSK 0
I20260926 13:18:46.818420 21695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:46.820989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21613
W20260926 13:18:46.825296 21598 connection.cc:535] client connection to 127.0.0.1:45025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45025): Connection reset by peer (error 104)
2026-09-26T13:18:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:46 dist-test-slave-f864 krb5kdc[21712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:46 dist-test-slave-f864 krb5kdc[21712](info): set up 2 sockets
Sep 26 13:18:46 dist-test-slave-f864 krb5kdc[21712](info): commencing operation
krb5kdc: starting...
W20260926 13:18:47.858381 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:47 dist-test-slave-f864 krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:47.981802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:48.065367 21725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:48.065584 21725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:48.065608 21725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:48.066826 21725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:48.066879 21725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:48.066893 21725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:48.066906 21725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:48.066918 21725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:48.068337 21725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:48.068524 21725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:48.068698 21725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 13:18:48.073537 21725 server_base.cc:1034] running on GCE node
W20260926 13:18:48.073683 21731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:48.073614 21730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:48.073614 21733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:48.074066 21725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:48.074415 21725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:48.075580 21725 hybrid_clock.cc:648] HybridClock initialized: now 1790428728075574 us; error 38 us; skew 500 ppm
Sep 26 13:18:48 dist-test-slave-f864 krb5kdc[21712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:48.078006 21725 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:48.078555 21725 webserver.cc:460] Webserver started at http://127.0.0.1:39663/ using document root <none> and password file <none>
I20260926 13:18:48.078804 21725 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:48.078871 21725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:48.079025 21725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:48.080097 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "436822e5245b4246a759270b7411122c"
format_stamp: "Formatted at 2026-09-26 13:18:48 on dist-test-slave-f864"
server_key: "43022504bbff349c9b8e618d262f6f19"
server_key_iv: "f1c818b9083aea0d42ff7d64527c4084"
server_key_version: "encryptionkey@0"
I20260926 13:18:48.080472 21725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "436822e5245b4246a759270b7411122c"
format_stamp: "Formatted at 2026-09-26 13:18:48 on dist-test-slave-f864"
server_key: "43022504bbff349c9b8e618d262f6f19"
server_key_iv: "f1c818b9083aea0d42ff7d64527c4084"
server_key_version: "encryptionkey@0"
I20260926 13:18:48.082021 21725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:48.083035 21740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:48.083223 21725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:48.083318 21725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "436822e5245b4246a759270b7411122c"
format_stamp: "Formatted at 2026-09-26 13:18:48 on dist-test-slave-f864"
server_key: "43022504bbff349c9b8e618d262f6f19"
server_key_iv: "f1c818b9083aea0d42ff7d64527c4084"
server_key_version: "encryptionkey@0"
I20260926 13:18:48.083400 21725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:48.107067 21725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:48.109040 21725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:48.109223 21725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:48.113322 21725 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36987
I20260926 13:18:48.113356 21792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36987 every 8 connection(s)
I20260926 13:18:48.113727 21725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:48.114795 21793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:48.116930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21725
I20260926 13:18:48.117059 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:48.117144 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c: Bootstrap starting.
I20260926 13:18:48.117264 392 external_mini_cluster.cc:1428] Setting key 69280f2e91d51eb6b1a44ba70c054533
I20260926 13:18:48.117936 21793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:48.118286 21793 log.cc:824] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:48.119014 21793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c: No bootstrap required, opened a new log
I20260926 13:18:48.121446 21793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436822e5245b4246a759270b7411122c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36987 } }
I20260926 13:18:48.121624 21793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:48.121665 21793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436822e5245b4246a759270b7411122c, State: Initialized, Role: FOLLOWER
I20260926 13:18:48.121791 21793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436822e5245b4246a759270b7411122c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36987 } }
I20260926 13:18:48.121863 21793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:48.121910 21793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:48.121958 21793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:48.122711 21793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436822e5245b4246a759270b7411122c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36987 } }
I20260926 13:18:48.122828 21793 leader_election.cc:302] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 436822e5245b4246a759270b7411122c; no voters:
I20260926 13:18:48.122994 21793 leader_election.cc:288] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:48.123056 21798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:48.123216 21798 raft_consensus.cc:686] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [term 1 LEADER]: Becoming Leader. State: Replica: 436822e5245b4246a759270b7411122c, State: Running, Role: LEADER
I20260926 13:18:48.123250 21793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:48.123310 21798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436822e5245b4246a759270b7411122c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36987 } }
I20260926 13:18:48.123896 21799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "436822e5245b4246a759270b7411122c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436822e5245b4246a759270b7411122c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36987 } } }
I20260926 13:18:48.124012 21799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:48.124068 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:48.117790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:46821 (local address 127.0.0.1:36987)
0926 13:18:48.117989 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:48.117999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:48.118033 (+ 34us) server_negotiation.cc:407] Connection header received
0926 13:18:48.118647 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:48.118655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:48.118742 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:48.118902 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:48.119069 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:48.120418 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:48.121610 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:48.122272 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:48.122354 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:48.123169 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:48.123174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:48.123174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:48.123501 (+ 327us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:48.123542 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:48.123558 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:48.123673 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:48.123777 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 13:18:48.123805 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260926 13:18:48.124145 21800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436822e5245b4246a759270b7411122c. Latest consensus state: current_term: 1 leader_uuid: "436822e5245b4246a759270b7411122c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436822e5245b4246a759270b7411122c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36987 } } }
I20260926 13:18:48.124200 21800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:48.124318 21807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:48.124814 21807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:48.124933 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:48.126493 21807 catalog_manager.cc:1347] Generated new cluster ID: 7626a7bbf9e84d01b68224aabada2bb8
I20260926 13:18:48.126549 21807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:48.133147 21796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:48.126404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:60851 (local address 127.0.0.1:36987)
0926 13:18:48.126446 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:48.126447 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:48.130348 (+ 3901us) server_negotiation.cc:407] Connection header received
0926 13:18:48.130422 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:48.130422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:48.130440 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:48.130507 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:48.130658 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:48.131139 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:48.131879 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:48.132100 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:48.132176 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:48.132864 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:48.132865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:48.132866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:48.132870 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:48.132884 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:48.132889 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:48.132965 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:48.133038 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 13:18:48.133048 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:18:48.169111 21807 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:48.169695 21807 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:48.174846 21807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 436822e5245b4246a759270b7411122c: Generated new TSK 0
I20260926 13:18:48.175058 21807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:48.177237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21725
W20260926 13:18:48.181262 21710 connection.cc:535] client connection to 127.0.0.1:36987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36987): Connection reset by peer (error 104)
2026-09-26T13:18:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:48 dist-test-slave-f864 krb5kdc[21824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:48 dist-test-slave-f864 krb5kdc[21824](info): set up 2 sockets
Sep 26 13:18:48 dist-test-slave-f864 krb5kdc[21824](info): commencing operation
krb5kdc: starting...
W20260926 13:18:49.222344 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:49 dist-test-slave-f864 krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 13:18:49.338953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:49.421186 21837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:49.421458 21837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:49.421499 21837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:49.422755 21837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:49.422832 21837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:49.422856 21837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:49.422878 21837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:49.422907 21837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:49.424444 21837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:49.424778 21837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:49.425067 21837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:49.430214 21842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:49.430150 21845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:49.430210 21843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:49.430435 21837 server_base.cc:1034] running on GCE node
I20260926 13:18:49.430691 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:49.430948 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:49.432128 21837 hybrid_clock.cc:648] HybridClock initialized: now 1790428729432119 us; error 47 us; skew 500 ppm
Sep 26 13:18:49 dist-test-slave-f864 krb5kdc[21824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:49.434810 21837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 13:18:49.435319 21837 webserver.cc:460] Webserver started at http://127.0.0.1:40981/ using document root <none> and password file <none>
I20260926 13:18:49.435560 21837 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:49.435631 21837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:49.435791 21837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:49.436905 21837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "bcf88021f84f4192866ab0b72107e923"
format_stamp: "Formatted at 2026-09-26 13:18:49 on dist-test-slave-f864"
server_key: "8263e91092a075048a061ca706800498"
server_key_iv: "df0568d30501294b49e46ade3d78d00a"
server_key_version: "encryptionkey@0"
I20260926 13:18:49.437288 21837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "bcf88021f84f4192866ab0b72107e923"
format_stamp: "Formatted at 2026-09-26 13:18:49 on dist-test-slave-f864"
server_key: "8263e91092a075048a061ca706800498"
server_key_iv: "df0568d30501294b49e46ade3d78d00a"
server_key_version: "encryptionkey@0"
I20260926 13:18:49.438761 21837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:49.439669 21852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:49.439849 21837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:49.439954 21837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "bcf88021f84f4192866ab0b72107e923"
format_stamp: "Formatted at 2026-09-26 13:18:49 on dist-test-slave-f864"
server_key: "8263e91092a075048a061ca706800498"
server_key_iv: "df0568d30501294b49e46ade3d78d00a"
server_key_version: "encryptionkey@0"
I20260926 13:18:49.440050 21837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:49.443696 21837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:49.446118 21837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:49.446344 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:49.450552 21837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35261
I20260926 13:18:49.450591 21904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35261 every 8 connection(s)
I20260926 13:18:49.450981 21837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:49.452085 21905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:49.453260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21837
I20260926 13:18:49.453395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:49.453563 392 external_mini_cluster.cc:1428] Setting key a849c33ab88a5f2ea02c368d2caa2eb2
I20260926 13:18:49.455109 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923: Bootstrap starting.
I20260926 13:18:49.455757 21905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:49.456092 21905 log.cc:824] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:49.456851 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923: No bootstrap required, opened a new log
I20260926 13:18:49.458470 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:49.454240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:41159 (local address 127.0.0.1:35261)
0926 13:18:49.456384 (+ 2144us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:49.456392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:49.456421 (+ 29us) server_negotiation.cc:407] Connection header received
0926 13:18:49.456922 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:49.456932 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:49.456989 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:49.457164 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:49.457990 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:49.457994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:49.457994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:49.457998 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:49.458044 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:49.458055 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:49.458156 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:49.458255 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:18:49.458303 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2093,"thread_start_us":92,"threads_started":1}
I20260926 13:18:49.459376 21905 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf88021f84f4192866ab0b72107e923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260926 13:18:49.459568 21905 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:49.459609 21905 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf88021f84f4192866ab0b72107e923, State: Initialized, Role: FOLLOWER
I20260926 13:18:49.459597 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:49.459745 21905 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf88021f84f4192866ab0b72107e923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260926 13:18:49.459826 21905 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:49.459870 21905 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:49.459923 21905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:49.460546 21905 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf88021f84f4192866ab0b72107e923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260926 13:18:49.460678 21905 leader_election.cc:302] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcf88021f84f4192866ab0b72107e923; no voters:
I20260926 13:18:49.460891 21905 leader_election.cc:288] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:49.460942 21914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:49.461104 21914 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [term 1 LEADER]: Becoming Leader. State: Replica: bcf88021f84f4192866ab0b72107e923, State: Running, Role: LEADER
I20260926 13:18:49.461221 21905 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:49.461313 21914 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf88021f84f4192866ab0b72107e923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } }
I20260926 13:18:49.461678 21914 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcf88021f84f4192866ab0b72107e923. Latest consensus state: current_term: 1 leader_uuid: "bcf88021f84f4192866ab0b72107e923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf88021f84f4192866ab0b72107e923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } } }
I20260926 13:18:49.461700 21915 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcf88021f84f4192866ab0b72107e923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf88021f84f4192866ab0b72107e923" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35261 } } }
I20260926 13:18:49.461783 21914 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:49.461789 21915 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:49.462057 21923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:49.462657 21923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:49.463045 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:49.461276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.103:55547 (local address 127.0.0.1:35261)
0926 13:18:49.461381 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:49.461384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:49.461567 (+ 183us) server_negotiation.cc:407] Connection header received
0926 13:18:49.461841 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:49.461843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:49.461849 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:49.461884 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:49.462733 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:49.462735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:49.462735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:49.462739 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:49.462757 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:49.462761 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:49.462828 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:49.462961 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 13:18:49.462975 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 13:18:49.464488 21923 catalog_manager.cc:1347] Generated new cluster ID: 2777e52c710e4c06886d965004abeafe
I20260926 13:18:49.464540 21923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:49.473151 21923 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:49.473345 21923 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:49.482043 21923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcf88021f84f4192866ab0b72107e923: Generated new TSK 0
I20260926 13:18:49.482316 21923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:49.490600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21837
2026-09-26T13:18:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:49 dist-test-slave-f864 krb5kdc[21936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:49 dist-test-slave-f864 krb5kdc[21936](info): set up 2 sockets
Sep 26 13:18:49 dist-test-slave-f864 krb5kdc[21936](info): commencing operation
krb5kdc: starting...
W20260926 13:18:50.606220 392 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:50 dist-test-slave-f864 krb5kdc[21936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:50.730453 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:50.817234 21949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:50.817543 21949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:50.817595 21949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:50.818898 21949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:50.818974 21949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:50.818998 21949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:50.819023 21949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:50.819052 21949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:50.821051 21949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:50.821364 21949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:50.821610 21949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:50.826750 21957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:50.826797 21955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:50.826750 21954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:50.827067 21949 server_base.cc:1034] running on GCE node
I20260926 13:18:50.827280 21949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:50.827553 21949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:50.828779 21949 hybrid_clock.cc:648] HybridClock initialized: now 1790428730828756 us; error 36 us; skew 500 ppm
Sep 26 13:18:50 dist-test-slave-f864 krb5kdc[21936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428730, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:50.837116 21949 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:50.837715 21949 webserver.cc:460] Webserver started at http://127.0.98.62:33987/ using document root <none> and password file <none>
I20260926 13:18:50.837970 21949 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:50.838048 21949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:50.838255 21949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:50.839457 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "7c5210316dd74c768f13eeaa4c0fc068"
format_stamp: "Formatted at 2026-09-26 13:18:50 on dist-test-slave-f864"
server_key: "626c0d125cbdb1a48a4c7480ae6c97c2"
server_key_iv: "4d9d43707dee7287849929b7e5e64649"
server_key_version: "encryptionkey@0"
I20260926 13:18:50.839864 21949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c5210316dd74c768f13eeaa4c0fc068"
format_stamp: "Formatted at 2026-09-26 13:18:50 on dist-test-slave-f864"
server_key: "626c0d125cbdb1a48a4c7480ae6c97c2"
server_key_iv: "4d9d43707dee7287849929b7e5e64649"
server_key_version: "encryptionkey@0"
I20260926 13:18:50.841310 21949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:50.842375 21964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:50.842583 21949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:50.842691 21949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "7c5210316dd74c768f13eeaa4c0fc068"
format_stamp: "Formatted at 2026-09-26 13:18:50 on dist-test-slave-f864"
server_key: "626c0d125cbdb1a48a4c7480ae6c97c2"
server_key_iv: "4d9d43707dee7287849929b7e5e64649"
server_key_version: "encryptionkey@0"
I20260926 13:18:50.842782 21949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:50.860447 21949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:50.863081 21949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:50.863250 21949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:50.867647 21949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44903
I20260926 13:18:50.867750 22016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44903 every 8 connection(s)
I20260926 13:18:50.868089 21949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:50.868981 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:50.871407 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068: Bootstrap starting.
I20260926 13:18:50.871953 22017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:50.872186 22017 log.cc:824] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:50.872870 22017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068: No bootstrap required, opened a new log
I20260926 13:18:50.874431 22017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5210316dd74c768f13eeaa4c0fc068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44903 } }
I20260926 13:18:50.874579 22017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:50.874603 22017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c5210316dd74c768f13eeaa4c0fc068, State: Initialized, Role: FOLLOWER
I20260926 13:18:50.874716 22017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5210316dd74c768f13eeaa4c0fc068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44903 } }
I20260926 13:18:50.874770 22017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:50.874791 22017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:50.874818 22017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:50.875311 22017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5210316dd74c768f13eeaa4c0fc068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44903 } }
I20260926 13:18:50.875394 22017 leader_election.cc:302] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c5210316dd74c768f13eeaa4c0fc068; no voters:
I20260926 13:18:50.875519 22017 leader_election.cc:288] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:50.875594 22020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:50.875792 22020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [term 1 LEADER]: Becoming Leader. State: Replica: 7c5210316dd74c768f13eeaa4c0fc068, State: Running, Role: LEADER
I20260926 13:18:50.875839 22017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:50.875927 22020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5210316dd74c768f13eeaa4c0fc068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44903 } }
I20260926 13:18:50.876078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 21949
I20260926 13:18:50.876209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:50.876407 392 external_mini_cluster.cc:1428] Setting key 4846273876979b8ea0665eaa8446bde8
I20260926 13:18:50.876652 22022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c5210316dd74c768f13eeaa4c0fc068. Latest consensus state: current_term: 1 leader_uuid: "7c5210316dd74c768f13eeaa4c0fc068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5210316dd74c768f13eeaa4c0fc068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44903 } } }
I20260926 13:18:50.876744 22022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:50.876799 22021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c5210316dd74c768f13eeaa4c0fc068" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c5210316dd74c768f13eeaa4c0fc068" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44903 } } }
I20260926 13:18:50.876868 22021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:50.877606 22029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:50.878141 22029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:50.879717 22029 catalog_manager.cc:1347] Generated new cluster ID: 59fa39589c5541fd98834e808f5dbb58
I20260926 13:18:50.879773 22029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:50 dist-test-slave-f864 krb5kdc[21936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:50.886343 22029 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:50.887045 22029 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:50.887427 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:50.877090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48345 (local address 127.0.98.62:44903)
0926 13:18:50.877241 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:50.877251 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:50.877386 (+ 135us) server_negotiation.cc:407] Connection header received
0926 13:18:50.877593 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:50.877601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:50.877661 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:50.877800 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:50.878616 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:50.879931 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:50.880719 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:50.881155 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:50.881285 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:50.883765 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:50.883773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:50.883773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:50.883784 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:50.885484 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:50.885981 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:50.885985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:50.885985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:50.886101 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:50.886397 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:50.886398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:50.886398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:50.886635 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:50.886833 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:50.886932 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 13:18:50.887184 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260926 13:18:50.888352 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:50.904902 22034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:50.890498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36385 (local address 127.0.98.62:44903)
0926 13:18:50.890541 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:50.890544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:50.898377 (+ 7833us) server_negotiation.cc:407] Connection header received
0926 13:18:50.898464 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:50.898465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:50.898476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:50.898551 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:50.899129 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:50.899707 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:50.900479 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:50.900706 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:50.900783 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:50.902078 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:50.902079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:50.902080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:50.902088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:50.903657 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:50.904042 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:50.904043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:50.904044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:50.904108 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:50.904360 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:50.904360 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:50.904361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:50.904476 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:50.904609 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:50.904718 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 13:18:50.904782 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:18:50.911141 22029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c5210316dd74c768f13eeaa4c0fc068: Generated new TSK 0
I20260926 13:18:50.911392 22029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:50.917596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 21949
W20260926 13:18:50.921908 21934 connection.cc:535] client connection to 127.0.98.62:44903 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44903): Connection reset by peer (error 104)
2026-09-26T13:18:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:50 dist-test-slave-f864 krb5kdc[22048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:50 dist-test-slave-f864 krb5kdc[22048](info): set up 2 sockets
Sep 26 13:18:50 dist-test-slave-f864 krb5kdc[22048](info): commencing operation
krb5kdc: starting...
W20260926 13:18:51.954465 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:52 dist-test-slave-f864 krb5kdc[22048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:52.081921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:52.166889 22061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:52.167100 22061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:52.167127 22061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:52.168290 22061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:52.168334 22061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:52.168347 22061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:52.168359 22061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:52.168375 22061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:52.169793 22061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:52.169991 22061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:52.170238 22061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:52.175181 22067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:52.175266 22069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:52.175355 22066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:52.175554 22061 server_base.cc:1034] running on GCE node
I20260926 13:18:52.175804 22061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:52.176051 22061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:52.177235 22061 hybrid_clock.cc:648] HybridClock initialized: now 1790428732177217 us; error 33 us; skew 500 ppm
Sep 26 13:18:52 dist-test-slave-f864 krb5kdc[22048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:52.179816 22061 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:52.180282 22061 webserver.cc:460] Webserver started at http://127.0.98.62:35193/ using document root <none> and password file <none>
I20260926 13:18:52.180528 22061 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:52.180574 22061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:52.180737 22061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:52.181798 22061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "679f09bf5fe74d8fbcbe6b12d174c03f"
format_stamp: "Formatted at 2026-09-26 13:18:52 on dist-test-slave-f864"
server_key: "ed3adcf278e0d21050450d6e09aabbb2"
server_key_iv: "caea4bf1d556a9745cf55052782ba882"
server_key_version: "encryptionkey@0"
I20260926 13:18:52.182147 22061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "679f09bf5fe74d8fbcbe6b12d174c03f"
format_stamp: "Formatted at 2026-09-26 13:18:52 on dist-test-slave-f864"
server_key: "ed3adcf278e0d21050450d6e09aabbb2"
server_key_iv: "caea4bf1d556a9745cf55052782ba882"
server_key_version: "encryptionkey@0"
I20260926 13:18:52.183497 22061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:52.184180 22076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:52.184351 22061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:52.184449 22061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "679f09bf5fe74d8fbcbe6b12d174c03f"
format_stamp: "Formatted at 2026-09-26 13:18:52 on dist-test-slave-f864"
server_key: "ed3adcf278e0d21050450d6e09aabbb2"
server_key_iv: "caea4bf1d556a9745cf55052782ba882"
server_key_version: "encryptionkey@0"
I20260926 13:18:52.184531 22061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:52.193655 22061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:52.195933 22061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:52.196091 22061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:52.200073 22061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40325
I20260926 13:18:52.200105 22128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40325 every 8 connection(s)
I20260926 13:18:52.200470 22061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:52.201287 22129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:52.203474 22129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f: Bootstrap starting.
I20260926 13:18:52.203971 22129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:52.204196 22129 log.cc:824] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:52.204803 22129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f: No bootstrap required, opened a new log
I20260926 13:18:52.206372 22129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260926 13:18:52.206529 22129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:52.206573 22129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679f09bf5fe74d8fbcbe6b12d174c03f, State: Initialized, Role: FOLLOWER
I20260926 13:18:52.206684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22061
I20260926 13:18:52.206725 22129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260926 13:18:52.206806 22129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:52.206809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:52.206840 22129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:52.206884 22129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:52.207005 392 external_mini_cluster.cc:1428] Setting key c710f6d852caf83a7a6f274423809198
I20260926 13:18:52.207572 22129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260926 13:18:52.207726 22129 leader_election.cc:302] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 679f09bf5fe74d8fbcbe6b12d174c03f; no voters:
I20260926 13:18:52.207911 22129 leader_election.cc:288] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:52.207932 22134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:52.208039 22134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [term 1 LEADER]: Becoming Leader. State: Replica: 679f09bf5fe74d8fbcbe6b12d174c03f, State: Running, Role: LEADER
I20260926 13:18:52.208165 22129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:52.208173 22134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } }
I20260926 13:18:52.208604 22136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 679f09bf5fe74d8fbcbe6b12d174c03f. Latest consensus state: current_term: 1 leader_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } } }
I20260926 13:18:52.208709 22136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:52.208961 22135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "679f09bf5fe74d8fbcbe6b12d174c03f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40325 } } }
I20260926 13:18:52.209084 22135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:52.209240 22142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:52.209781 22142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:52.212220 22142 catalog_manager.cc:1347] Generated new cluster ID: d701a1c379b2409da1e7b6f8fd4f50da
I20260926 13:18:52.212275 22142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:52.213444 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:52.207481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36407 (local address 127.0.98.62:40325)
0926 13:18:52.207667 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:52.207677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:52.207713 (+ 36us) server_negotiation.cc:407] Connection header received
0926 13:18:52.207824 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:52.207846 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:52.207901 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:52.208300 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:52.208391 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:52.210239 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:52.211329 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:52.211714 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:52.211795 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:52.212590 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:52.212594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:52.212594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:52.212820 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:52.212857 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:52.212870 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:52.212950 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:52.213199 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 13:18:52.213227 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260926 13:18:52.214551 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:52.219931 22133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:52.216011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34343 (local address 127.0.98.62:40325)
0926 13:18:52.216047 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:52.216049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:52.216154 (+ 105us) server_negotiation.cc:407] Connection header received
0926 13:18:52.216243 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:52.216244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:52.216253 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:52.216282 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:52.216462 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:52.217449 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:52.218529 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:52.218847 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:52.218888 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:52.219636 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:52.219637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:52.219638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:52.219642 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:52.219656 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:52.219660 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:52.219692 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:52.219842 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 13:18:52.219855 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:18:52.221204 22142 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:52.221695 22142 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:52.229539 22142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 679f09bf5fe74d8fbcbe6b12d174c03f: Generated new TSK 0
I20260926 13:18:52.229722 22142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:52.234833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22061
W20260926 13:18:52.238897 22046 connection.cc:535] client connection to 127.0.98.62:40325 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40325): Connection reset by peer (error 104)
2026-09-26T13:18:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:52 dist-test-slave-f864 krb5kdc[22160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:52 dist-test-slave-f864 krb5kdc[22160](info): set up 2 sockets
Sep 26 13:18:52 dist-test-slave-f864 krb5kdc[22160](info): commencing operation
krb5kdc: starting...
W20260926 13:18:53.270285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:53 dist-test-slave-f864 krb5kdc[22160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428733, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:53.389782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34687
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:53.470633 22173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:53.470842 22173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:53.470865 22173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:53.472008 22173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:53.472050 22173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:53.472064 22173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:53.472076 22173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:53.472090 22173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:53.473563 22173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34687
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:53.473841 22173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:53.474084 22173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:53.478833 22181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:53.478984 22173 server_base.cc:1034] running on GCE node
W20260926 13:18:53.478842 22179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:53.478871 22178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:53.479336 22173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:53.479650 22173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:53.480814 22173 hybrid_clock.cc:648] HybridClock initialized: now 1790428733480792 us; error 35 us; skew 500 ppm
Sep 26 13:18:53 dist-test-slave-f864 krb5kdc[22160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:53.483325 22173 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:53.483786 22173 webserver.cc:460] Webserver started at http://127.0.98.62:37169/ using document root <none> and password file <none>
I20260926 13:18:53.484016 22173 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:53.484086 22173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:53.484238 22173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:53.485275 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "d087268841be4da2a36d73f11d46a1c2"
format_stamp: "Formatted at 2026-09-26 13:18:53 on dist-test-slave-f864"
server_key: "4fd731ba2438dcda941f2ea2aacdf952"
server_key_iv: "2963c93ff81ebe4198d9a2f366b7c05b"
server_key_version: "encryptionkey@0"
I20260926 13:18:53.485636 22173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "d087268841be4da2a36d73f11d46a1c2"
format_stamp: "Formatted at 2026-09-26 13:18:53 on dist-test-slave-f864"
server_key: "4fd731ba2438dcda941f2ea2aacdf952"
server_key_iv: "2963c93ff81ebe4198d9a2f366b7c05b"
server_key_version: "encryptionkey@0"
I20260926 13:18:53.487047 22173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:53.487860 22188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:53.488061 22173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:53.488168 22173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "d087268841be4da2a36d73f11d46a1c2"
format_stamp: "Formatted at 2026-09-26 13:18:53 on dist-test-slave-f864"
server_key: "4fd731ba2438dcda941f2ea2aacdf952"
server_key_iv: "2963c93ff81ebe4198d9a2f366b7c05b"
server_key_version: "encryptionkey@0"
I20260926 13:18:53.488268 22173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:53.491608 22173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:53.493455 22173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:53.493609 22173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:53.497637 22173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34687
I20260926 13:18:53.497664 22240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34687 every 8 connection(s)
I20260926 13:18:53.497992 22173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:53.498989 22241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:53.501123 22241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2: Bootstrap starting.
I20260926 13:18:53.501639 22241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:53.501869 22241 log.cc:824] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:53.502557 22241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2: No bootstrap required, opened a new log
I20260926 13:18:53.504037 22241 raft_consensus.cc:348] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d087268841be4da2a36d73f11d46a1c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34687 } }
I20260926 13:18:53.504171 22241 raft_consensus.cc:374] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:53.504194 22241 raft_consensus.cc:729] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d087268841be4da2a36d73f11d46a1c2, State: Initialized, Role: FOLLOWER
I20260926 13:18:53.504336 22241 consensus_queue.cc:260] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d087268841be4da2a36d73f11d46a1c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34687 } }
I20260926 13:18:53.504413 22241 raft_consensus.cc:388] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:53.504436 22241 raft_consensus.cc:482] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:53.504464 22241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:53.504884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22173
I20260926 13:18:53.505012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:53.505106 22241 raft_consensus.cc:504] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d087268841be4da2a36d73f11d46a1c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34687 } }
I20260926 13:18:53.505198 392 external_mini_cluster.cc:1428] Setting key 65fd1b900e12f6f0be35048880e7d378
I20260926 13:18:53.505213 22241 leader_election.cc:302] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d087268841be4da2a36d73f11d46a1c2; no voters:
I20260926 13:18:53.505389 22241 leader_election.cc:288] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:53.505443 22244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:53.505628 22244 raft_consensus.cc:686] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [term 1 LEADER]: Becoming Leader. State: Replica: d087268841be4da2a36d73f11d46a1c2, State: Running, Role: LEADER
I20260926 13:18:53.505651 22241 sys_catalog.cc:564] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:53.505995 22244 consensus_queue.cc:237] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d087268841be4da2a36d73f11d46a1c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34687 } }
I20260926 13:18:53.506839 22245 sys_catalog.cc:455] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d087268841be4da2a36d73f11d46a1c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d087268841be4da2a36d73f11d46a1c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34687 } } }
I20260926 13:18:53.506935 22245 sys_catalog.cc:458] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:53.507169 22256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:53.507507 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d087268841be4da2a36d73f11d46a1c2. Latest consensus state: current_term: 1 leader_uuid: "d087268841be4da2a36d73f11d46a1c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d087268841be4da2a36d73f11d46a1c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34687 } } }
I20260926 13:18:53.507644 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:53.507756 22256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:53.508433 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:53.505815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57367 (local address 127.0.98.62:34687)
0926 13:18:53.506010 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:53.506019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:53.506079 (+ 60us) server_negotiation.cc:407] Connection header received
0926 13:18:53.506238 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:53.506248 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:53.506287 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:53.506431 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:53.507896 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:53.507900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:53.507900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:53.507904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:53.507973 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:53.507990 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:53.508063 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:53.508231 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 13:18:53.508272 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260926 13:18:53.509336 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:53.509404 22256 catalog_manager.cc:1347] Generated new cluster ID: 355e40d78f394bf7a0893764d03d16f0
I20260926 13:18:53.509438 22256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:53.511898 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:53.510867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49671 (local address 127.0.98.62:34687)
0926 13:18:53.510902 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:53.510904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:53.511036 (+ 132us) server_negotiation.cc:407] Connection header received
0926 13:18:53.511096 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:53.511097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:53.511102 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:53.511147 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:53.511703 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:53.511704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:53.511704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:53.511707 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:53.511720 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:53.511724 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:53.511773 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:53.511828 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:18:53.511840 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 13:18:53.522935 22256 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:53.523149 22256 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:53.539434 22256 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d087268841be4da2a36d73f11d46a1c2: Generated new TSK 0
I20260926 13:18:53.539665 22256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:53.543617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22173
2026-09-26T13:18:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:53 dist-test-slave-f864 krb5kdc[22272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:53 dist-test-slave-f864 krb5kdc[22272](info): set up 2 sockets
Sep 26 13:18:53 dist-test-slave-f864 krb5kdc[22272](info): commencing operation
krb5kdc: starting...
W20260926 13:18:54.590286 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:54 dist-test-slave-f864 krb5kdc[22272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:54.707513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:54.789937 22285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:54.790230 22285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:54.790280 22285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:54.791389 22285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:54.791432 22285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:54.791448 22285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:54.791497 22285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:54.791529 22285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:54.792986 22285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:54.793191 22285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:54.793373 22285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:54.798094 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:54.798080 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:54.798105 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:54.798384 22285 server_base.cc:1034] running on GCE node
I20260926 13:18:54.798604 22285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:54.798935 22285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:54.800077 22285 hybrid_clock.cc:648] HybridClock initialized: now 1790428734800054 us; error 40 us; skew 500 ppm
Sep 26 13:18:54 dist-test-slave-f864 krb5kdc[22272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:54.802646 22285 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:54.803077 22285 webserver.cc:460] Webserver started at http://127.0.98.62:37115/ using document root <none> and password file <none>
I20260926 13:18:54.803315 22285 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:54.803359 22285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:54.803486 22285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:54.804525 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "9467b056a4834196914cba34504ce417"
format_stamp: "Formatted at 2026-09-26 13:18:54 on dist-test-slave-f864"
server_key: "3cd856cf76a5d2612397ad2722bfb46b"
server_key_iv: "77f3721093f57307d80b12891c536aad"
server_key_version: "encryptionkey@0"
I20260926 13:18:54.804904 22285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "9467b056a4834196914cba34504ce417"
format_stamp: "Formatted at 2026-09-26 13:18:54 on dist-test-slave-f864"
server_key: "3cd856cf76a5d2612397ad2722bfb46b"
server_key_iv: "77f3721093f57307d80b12891c536aad"
server_key_version: "encryptionkey@0"
I20260926 13:18:54.806282 22285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:18:54.807085 22300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:54.807286 22285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 13:18:54.807355 22285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "9467b056a4834196914cba34504ce417"
format_stamp: "Formatted at 2026-09-26 13:18:54 on dist-test-slave-f864"
server_key: "3cd856cf76a5d2612397ad2722bfb46b"
server_key_iv: "77f3721093f57307d80b12891c536aad"
server_key_version: "encryptionkey@0"
I20260926 13:18:54.807446 22285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:54.817657 22285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:54.820043 22285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:54.820210 22285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:54.824400 22285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37419
I20260926 13:18:54.824429 22352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37419 every 8 connection(s)
I20260926 13:18:54.824826 22285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:54.825697 22353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:54.827976 22353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417: Bootstrap starting.
I20260926 13:18:54.828544 22353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:54.828828 22353 log.cc:824] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:54.829487 22353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417: No bootstrap required, opened a new log
I20260926 13:18:54.831148 22353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9467b056a4834196914cba34504ce417" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260926 13:18:54.831302 22353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:54.831326 22353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9467b056a4834196914cba34504ce417, State: Initialized, Role: FOLLOWER
I20260926 13:18:54.831522 22353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9467b056a4834196914cba34504ce417" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260926 13:18:54.831585 22353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:54.831609 22353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:54.831676 22353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:54.832216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22285
I20260926 13:18:54.832274 22353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9467b056a4834196914cba34504ce417" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260926 13:18:54.832340 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:54.832386 22353 leader_election.cc:302] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9467b056a4834196914cba34504ce417; no voters:
I20260926 13:18:54.832528 392 external_mini_cluster.cc:1428] Setting key 16f27ce55c8ff84b09bd870d08959e41
I20260926 13:18:54.832547 22353 leader_election.cc:288] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:54.832614 22356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:54.832793 22353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:54.832794 22356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [term 1 LEADER]: Becoming Leader. State: Replica: 9467b056a4834196914cba34504ce417, State: Running, Role: LEADER
I20260926 13:18:54.833042 22356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9467b056a4834196914cba34504ce417" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } }
I20260926 13:18:54.833570 22357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9467b056a4834196914cba34504ce417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9467b056a4834196914cba34504ce417" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260926 13:18:54.833657 22358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9467b056a4834196914cba34504ce417. Latest consensus state: current_term: 1 leader_uuid: "9467b056a4834196914cba34504ce417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9467b056a4834196914cba34504ce417" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37419 } } }
I20260926 13:18:54.833731 22357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:54.833976 22358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:54.834153 22368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:54.834774 22368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:54.837046 22368 catalog_manager.cc:1347] Generated new cluster ID: 3e53d7f3f98f4c9d892ea56053d5e0fa
I20260926 13:18:54.837096 22368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 13:18:54 dist-test-slave-f864 krb5kdc[22272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:54.844847 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:54.833499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33067 (local address 127.0.98.62:37419)
0926 13:18:54.833851 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:54.833861 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:54.833892 (+ 31us) server_negotiation.cc:407] Connection header received
0926 13:18:54.834019 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:54.834028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:54.834100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:54.835122 (+ 1022us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:54.835212 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:54.836569 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:54.837586 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:54.838266 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:54.838378 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:54.840988 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:54.840994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:54.840995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:54.841010 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:54.843032 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:54.843423 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:54.843426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:54.843426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:54.843552 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:54.843755 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:54.843756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:54.843757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:54.844082 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:54.844252 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:54.844327 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 13:18:54.844602 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":86,"threads_started":1}
I20260926 13:18:54.845853 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:54.854318 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:54.847250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43449 (local address 127.0.98.62:37419)
0926 13:18:54.847291 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:54.847293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:54.847376 (+ 83us) server_negotiation.cc:407] Connection header received
0926 13:18:54.847420 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:54.847420 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:54.847431 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:54.847495 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:54.848029 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:54.848801 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:54.849752 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:54.850065 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:54.850137 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:54.851486 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:54.851487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:54.851488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:54.851494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:54.853084 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:54.853453 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:54.853454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:54.853454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:54.853512 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:54.853714 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:54.853715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:54.853715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:54.853841 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:54.853937 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:54.854035 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 13:18:54.854110 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:18:54.866950 22368 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:54.867585 22368 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:54.872491 22368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9467b056a4834196914cba34504ce417: Generated new TSK 0
I20260926 13:18:54.872702 22368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:54.879088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22285
2026-09-26T13:18:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:54 dist-test-slave-f864 krb5kdc[22384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:54 dist-test-slave-f864 krb5kdc[22384](info): set up 2 sockets
Sep 26 13:18:54 dist-test-slave-f864 krb5kdc[22384](info): commencing operation
krb5kdc: starting...
W20260926 13:18:55.914361 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:55 dist-test-slave-f864 krb5kdc[22384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:56.032958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:56.112694 22397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:56.112902 22397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:56.112926 22397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:56.114552 22397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:56.114619 22397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:56.114643 22397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:56.114674 22397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:56.114701 22397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:56.116658 22397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:56.116953 22397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:56.117198 22397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:56.122229 22402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:56.122234 22403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:56.122258 22405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:56.122264 22397 server_base.cc:1034] running on GCE node
I20260926 13:18:56.122656 22397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:56.122956 22397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:56.124101 22397 hybrid_clock.cc:648] HybridClock initialized: now 1790428736124090 us; error 28 us; skew 500 ppm
Sep 26 13:18:56 dist-test-slave-f864 krb5kdc[22384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:56.133435 22397 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:56.133941 22397 webserver.cc:460] Webserver started at http://127.0.98.62:42261/ using document root <none> and password file <none>
I20260926 13:18:56.134222 22397 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:56.134308 22397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:56.134466 22397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:56.135561 22397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "00259d16971f4d8e927c9c73ce188482"
format_stamp: "Formatted at 2026-09-26 13:18:56 on dist-test-slave-f864"
server_key: "938407bebbdd6ca225c9829a44f36589"
server_key_iv: "95a7a7848289185cfcd425f3b69165f1"
server_key_version: "encryptionkey@0"
I20260926 13:18:56.135941 22397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "00259d16971f4d8e927c9c73ce188482"
format_stamp: "Formatted at 2026-09-26 13:18:56 on dist-test-slave-f864"
server_key: "938407bebbdd6ca225c9829a44f36589"
server_key_iv: "95a7a7848289185cfcd425f3b69165f1"
server_key_version: "encryptionkey@0"
I20260926 13:18:56.137362 22397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 13:18:56.138085 22412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:56.138284 22397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:56.138387 22397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "00259d16971f4d8e927c9c73ce188482"
format_stamp: "Formatted at 2026-09-26 13:18:56 on dist-test-slave-f864"
server_key: "938407bebbdd6ca225c9829a44f36589"
server_key_iv: "95a7a7848289185cfcd425f3b69165f1"
server_key_version: "encryptionkey@0"
I20260926 13:18:56.138461 22397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:56.157565 22397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:56.159677 22397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:56.159835 22397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:56.163807 22397 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34255
I20260926 13:18:56.163841 22464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34255 every 8 connection(s)
I20260926 13:18:56.164216 22397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:56.165042 22465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:56.167284 22465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482: Bootstrap starting.
I20260926 13:18:56.167730 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22397
I20260926 13:18:56.167836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:56.167932 22465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:56.168056 392 external_mini_cluster.cc:1428] Setting key b9ae2d9491f746880fe3a8b06ed94fa3
I20260926 13:18:56.168265 22465 log.cc:824] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:56.168998 22465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482: No bootstrap required, opened a new log
I20260926 13:18:56.171561 22465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00259d16971f4d8e927c9c73ce188482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 13:18:56.171720 22465 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:56.171764 22465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00259d16971f4d8e927c9c73ce188482, State: Initialized, Role: FOLLOWER
I20260926 13:18:56.171885 22465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00259d16971f4d8e927c9c73ce188482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 13:18:56.171980 22465 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:56.172024 22465 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:56.172093 22465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:56.172735 22465 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00259d16971f4d8e927c9c73ce188482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 13:18:56.172843 22465 leader_election.cc:302] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00259d16971f4d8e927c9c73ce188482; no voters:
I20260926 13:18:56.173003 22465 leader_election.cc:288] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:56.173110 22470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:56.173266 22470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [term 1 LEADER]: Becoming Leader. State: Replica: 00259d16971f4d8e927c9c73ce188482, State: Running, Role: LEADER
I20260926 13:18:56.173382 22470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00259d16971f4d8e927c9c73ce188482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } }
I20260926 13:18:56.173614 22465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:56.173712 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00259d16971f4d8e927c9c73ce188482. Latest consensus state: current_term: 1 leader_uuid: "00259d16971f4d8e927c9c73ce188482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00259d16971f4d8e927c9c73ce188482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } } }
I20260926 13:18:56.173784 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:56.173969 22472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00259d16971f4d8e927c9c73ce188482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00259d16971f4d8e927c9c73ce188482" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34255 } } }
I20260926 13:18:56.174085 22472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:56.174465 22474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:56.174472 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:56.168648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33827 (local address 127.0.98.62:34255)
0926 13:18:56.168802 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:56.168812 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:56.168844 (+ 32us) server_negotiation.cc:407] Connection header received
0926 13:18:56.169385 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:56.169394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:56.169456 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:56.169608 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:56.169812 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:56.171240 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:56.172071 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:56.172693 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:56.172807 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:56.173544 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:56.173548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:56.173548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:56.173758 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:56.173797 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:56.173814 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:56.173879 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:56.174136 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 13:18:56.174230 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":81,"threads_started":1}
I20260926 13:18:56.174979 22474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:56.175635 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:56.176491 22474 catalog_manager.cc:1347] Generated new cluster ID: 8dc302f875424e248f53ee46cf76b1d3
I20260926 13:18:56.176540 22474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:56.180354 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:56.177242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50697 (local address 127.0.98.62:34255)
0926 13:18:56.177274 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:56.177276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:56.177286 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:18:56.177320 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:56.177320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:56.177329 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:56.177385 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:56.177478 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:56.178205 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:56.179150 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:56.179451 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:56.179522 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:56.180140 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:56.180141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:56.180141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:56.180145 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:56.180159 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:56.180163 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:56.180219 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:56.180269 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 13:18:56.180288 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:18:56.185503 22474 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:56.185992 22474 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:56.188717 22474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00259d16971f4d8e927c9c73ce188482: Generated new TSK 0
I20260926 13:18:56.188889 22474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:56.191740 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22397
2026-09-26T13:18:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:56 dist-test-slave-f864 krb5kdc[22496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:56 dist-test-slave-f864 krb5kdc[22496](info): set up 2 sockets
Sep 26 13:18:56 dist-test-slave-f864 krb5kdc[22496](info): commencing operation
krb5kdc: starting...
W20260926 13:18:57.218379 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:57 dist-test-slave-f864 krb5kdc[22496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:57.333554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:57.414614 22509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:57.414816 22509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:57.414839 22509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:57.415911 22509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:57.415946 22509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:57.415961 22509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:57.415972 22509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:57.415984 22509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:57.417439 22509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:57.417631 22509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:57.417837 22509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:57.422591 22514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:57.422578 22517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:57.422577 22515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:57.422801 22509 server_base.cc:1034] running on GCE node
I20260926 13:18:57.423048 22509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:57.423358 22509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:57.424523 22509 hybrid_clock.cc:648] HybridClock initialized: now 1790428737424501 us; error 40 us; skew 500 ppm
Sep 26 13:18:57 dist-test-slave-f864 krb5kdc[22496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428737, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:57.426928 22509 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:57.427357 22509 webserver.cc:460] Webserver started at http://127.0.98.62:36033/ using document root <none> and password file <none>
I20260926 13:18:57.427590 22509 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:57.427660 22509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:57.427809 22509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:57.428821 22509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "37ffa453fe85429bb5a70da0a06ebe35"
format_stamp: "Formatted at 2026-09-26 13:18:57 on dist-test-slave-f864"
server_key: "42b29bcb936204a12c83fa7cc7a24a27"
server_key_iv: "2e3be37944579ac3d7d28d89e1ff2a94"
server_key_version: "encryptionkey@0"
I20260926 13:18:57.429180 22509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "37ffa453fe85429bb5a70da0a06ebe35"
format_stamp: "Formatted at 2026-09-26 13:18:57 on dist-test-slave-f864"
server_key: "42b29bcb936204a12c83fa7cc7a24a27"
server_key_iv: "2e3be37944579ac3d7d28d89e1ff2a94"
server_key_version: "encryptionkey@0"
I20260926 13:18:57.430506 22509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:57.431214 22524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:57.431387 22509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:57.431490 22509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "37ffa453fe85429bb5a70da0a06ebe35"
format_stamp: "Formatted at 2026-09-26 13:18:57 on dist-test-slave-f864"
server_key: "42b29bcb936204a12c83fa7cc7a24a27"
server_key_iv: "2e3be37944579ac3d7d28d89e1ff2a94"
server_key_version: "encryptionkey@0"
I20260926 13:18:57.431562 22509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:57.434847 22509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:57.436604 22509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:57.436753 22509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:57.440529 22509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35175
I20260926 13:18:57.440563 22576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35175 every 8 connection(s)
I20260926 13:18:57.440863 22509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:57.441766 22577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:57.443962 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35: Bootstrap starting.
I20260926 13:18:57.444509 22577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:57.444799 22577 log.cc:824] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:57.445449 22577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35: No bootstrap required, opened a new log
I20260926 13:18:57.447057 22577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ffa453fe85429bb5a70da0a06ebe35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35175 } }
I20260926 13:18:57.447199 22577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 13:18:57.447222 22577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ffa453fe85429bb5a70da0a06ebe35, State: Initialized, Role: FOLLOWER
I20260926 13:18:57.447367 22577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ffa453fe85429bb5a70da0a06ebe35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35175 } }
I20260926 13:18:57.447428 22577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 13:18:57.447449 22577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 13:18:57.447496 22577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:57.447996 22577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ffa453fe85429bb5a70da0a06ebe35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35175 } }
I20260926 13:18:57.448082 22577 leader_election.cc:302] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37ffa453fe85429bb5a70da0a06ebe35; no voters:
I20260926 13:18:57.448259 22577 leader_election.cc:288] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 13:18:57.448359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22509
I20260926 13:18:57.448432 22580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:57.448465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:57.448523 22577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:57.448625 22580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [term 1 LEADER]: Becoming Leader. State: Replica: 37ffa453fe85429bb5a70da0a06ebe35, State: Running, Role: LEADER
I20260926 13:18:57.448671 392 external_mini_cluster.cc:1428] Setting key 6898b1e1b9482e8b06a9d056ed88600d
I20260926 13:18:57.448760 22580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ffa453fe85429bb5a70da0a06ebe35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35175 } }
I20260926 13:18:57.449745 22581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37ffa453fe85429bb5a70da0a06ebe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ffa453fe85429bb5a70da0a06ebe35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35175 } } }
I20260926 13:18:57.449836 22581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:57.450239 22591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:18:57.450754 22591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:57.451231 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:57.449149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37413 (local address 127.0.98.62:35175)
0926 13:18:57.449423 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:57.449431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:57.449456 (+ 25us) server_negotiation.cc:407] Connection header received
0926 13:18:57.449592 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:57.449599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:57.449633 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:57.450026 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:57.450640 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:57.450644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:57.450644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:57.450648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:57.450691 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:57.450705 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:57.450791 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:57.451010 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 13:18:57.451052 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":198,"threads_started":1}
I20260926 13:18:57.451483 22582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37ffa453fe85429bb5a70da0a06ebe35. Latest consensus state: current_term: 1 leader_uuid: "37ffa453fe85429bb5a70da0a06ebe35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37ffa453fe85429bb5a70da0a06ebe35" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35175 } } }
I20260926 13:18:57.451556 22582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:57.452183 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:57.453773 22591 catalog_manager.cc:1347] Generated new cluster ID: 3905cd6acc564615adfd7577d7c95373
I20260926 13:18:57.453823 22591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:57.455170 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:57.453760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52531 (local address 127.0.98.62:35175)
0926 13:18:57.454238 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:57.454240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:57.454247 (+ 7us) server_negotiation.cc:407] Connection header received
0926 13:18:57.454272 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:57.454273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:57.454285 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:57.454350 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:57.454957 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:57.454958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:57.454959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 13:18:57.454962 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:57.454976 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 13:18:57.454980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:57.455029 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:57.455084 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 13:18:57.455096 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460}
I20260926 13:18:57.470031 22591 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:57.470247 22591 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:57.472834 22591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37ffa453fe85429bb5a70da0a06ebe35: Generated new TSK 0
I20260926 13:18:57.473006 22591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:18:57.478992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22509
2026-09-26T13:18:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1285 ms)
[----------] 18 tests from AuthTokenIssuingTest (23857 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:18:57 dist-test-slave-f864 krb5kdc[22608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:18:57 dist-test-slave-f864 krb5kdc[22608](info): set up 2 sockets
Sep 26 13:18:57 dist-test-slave-f864 krb5kdc[22608](info): commencing operation
krb5kdc: starting...
W20260926 13:18:58.502379 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:18:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:18:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:18:58.620685 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33087
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33087,127.0.98.61:33411,127.0.98.60:35075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:18:58.700537 22621 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:18:58.700785 22621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:58.700814 22621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:58.700855 22621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:58.701980 22621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:58.702040 22621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:58.702064 22621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:58.702086 22621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:58.702117 22621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:58.703613 22621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33087,127.0.98.61:33411,127.0.98.60:35075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:58.703883 22621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:58.704102 22621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:58.708849 22626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:58.708902 22621 server_base.cc:1034] running on GCE node
W20260926 13:18:58.708858 22629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:58.708865 22627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:58.709292 22621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:58.709610 22621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:58.710770 22621 hybrid_clock.cc:648] HybridClock initialized: now 1790428738710765 us; error 39 us; skew 500 ppm
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:58.713271 22621 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:18:58.713718 22621 webserver.cc:460] Webserver started at http://127.0.98.62:40009/ using document root <none> and password file <none>
I20260926 13:18:58.713904 22621 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:58.713943 22621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:58.714032 22621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:58.715075 22621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "d9cb9b6177e7426bb99625e728c77ae2"
format_stamp: "Formatted at 2026-09-26 13:18:58 on dist-test-slave-f864"
server_key: "4225bfb60e62857880cb1ecae6bb3685"
server_key_iv: "065dd1f50344335e81c12ca69b870272"
server_key_version: "encryptionkey@0"
I20260926 13:18:58.715366 22621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9cb9b6177e7426bb99625e728c77ae2"
format_stamp: "Formatted at 2026-09-26 13:18:58 on dist-test-slave-f864"
server_key: "4225bfb60e62857880cb1ecae6bb3685"
server_key_iv: "065dd1f50344335e81c12ca69b870272"
server_key_version: "encryptionkey@0"
I20260926 13:18:58.716593 22621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 13:18:58.717242 22636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:58.717388 22621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:18:58.717473 22621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "d9cb9b6177e7426bb99625e728c77ae2"
format_stamp: "Formatted at 2026-09-26 13:18:58 on dist-test-slave-f864"
server_key: "4225bfb60e62857880cb1ecae6bb3685"
server_key_iv: "065dd1f50344335e81c12ca69b870272"
server_key_version: "encryptionkey@0"
I20260926 13:18:58.717527 22621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:58.733270 22621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:58.735222 22621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:58.735338 22621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:58.739217 22621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33087
I20260926 13:18:58.739295 22688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33087 every 8 connection(s)
I20260926 13:18:58.739621 22621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:18:58.740736 22689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:58.742341 22689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:58.745728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22621
I20260926 13:18:58.745819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:18:58.746045 392 external_mini_cluster.cc:1428] Setting key 680f959c2448af52aae134e0cc911caf
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:58.752883 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.743067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.0.98.62:33087)
0926 13:18:58.743260 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:58.743263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:58.743293 (+ 30us) server_negotiation.cc:407] Connection header received
0926 13:18:58.743846 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:58.743848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:58.743865 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:58.743953 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:58.744524 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.745408 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.746572 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.747098 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.747221 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.749543 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:58.749550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:58.749550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:58.749557 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:58.751241 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.751653 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.751656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.751657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.751741 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.751968 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.751969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.751970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.752196 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:58.752293 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:58.752469 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 13:18:58.752620 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":119,"threads_started":1}
I20260926 13:18:58.752885 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.743015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33087 (local address 127.0.0.1:38492)
0926 13:18:58.743177 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.743223 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:58.743716 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:58.743977 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:58.743979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:58.744241 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:58.744426 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.744430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.745499 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.745500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.746490 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.746492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.747204 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.747205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.747256 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.747842 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:58.747853 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:58.749439 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:58.751347 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.751354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.751356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.751552 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.751818 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.751819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.751819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.751904 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.752270 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:58.752275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:58.752338 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 13:18:58.752448 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 13:18:58.752693 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":91,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260926 13:18:58.753680 22689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:58.754045 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.753932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41542)
0926 13:18:58.753981 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.754024 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:18:58.754217 22637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
I20260926 13:18:58.754648 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.754536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41546)
0926 13:18:58.754560 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.754607 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 13:18:58.754770 22689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33411: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 13:18:58.773501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33411
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33087,127.0.98.61:33411,127.0.98.60:35075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:18:58.804065 22689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } attempt: 1
I20260926 13:18:58.804600 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.804453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41548)
0926 13:18:58.804524 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.804570 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 13:18:58.805099 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.804957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41560)
0926 13:18:58.805033 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.805071 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 13:18:58.805253 22689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33411: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
W20260926 13:18:58.854241 22696 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:18:58.854494 22696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:58.854521 22696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:58.854600 22696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:58.855762 22696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:58.855836 22696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:58.855861 22696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:58.855893 22696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:58.855942 22696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:58.857435 22696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33087,127.0.98.61:33411,127.0.98.60:35075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:58.857728 22696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:58.857966 22696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:58.862946 22704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:58.862972 22701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:58.863116 22696 server_base.cc:1034] running on GCE node
W20260926 13:18:58.863181 22702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:58.863456 22696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:58.863766 22696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:58.864912 22696 hybrid_clock.cc:648] HybridClock initialized: now 1790428738864901 us; error 26 us; skew 500 ppm
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:58.867472 22696 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 13:18:58.867877 22696 webserver.cc:460] Webserver started at http://127.0.98.61:35997/ using document root <none> and password file <none>
I20260926 13:18:58.868110 22696 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:58.868191 22696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:58.868342 22696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:58.869406 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data/instance:
uuid: "05c6e8c521464c14adc5a7efc6ed19e2"
format_stamp: "Formatted at 2026-09-26 13:18:58 on dist-test-slave-f864"
server_key: "217f6837e5bdc773183bd9cb7d82e8f8"
server_key_iv: "2b9b2b4358086a58d8a9d8ac0e447c01"
server_key_version: "encryptionkey@0"
I20260926 13:18:58.869784 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal/instance:
uuid: "05c6e8c521464c14adc5a7efc6ed19e2"
format_stamp: "Formatted at 2026-09-26 13:18:58 on dist-test-slave-f864"
server_key: "217f6837e5bdc773183bd9cb7d82e8f8"
server_key_iv: "2b9b2b4358086a58d8a9d8ac0e447c01"
server_key_version: "encryptionkey@0"
I20260926 13:18:58.871177 22696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 13:18:58.871897 22711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:58.872066 22696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 13:18:58.872184 22696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal
uuid: "05c6e8c521464c14adc5a7efc6ed19e2"
format_stamp: "Formatted at 2026-09-26 13:18:58 on dist-test-slave-f864"
server_key: "217f6837e5bdc773183bd9cb7d82e8f8"
server_key_iv: "2b9b2b4358086a58d8a9d8ac0e447c01"
server_key_version: "encryptionkey@0"
I20260926 13:18:58.872258 22696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:58.873409 22689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } attempt: 2
I20260926 13:18:58.873787 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.873645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41576)
0926 13:18:58.873710 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.873751 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 13:18:58.874204 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.874070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41588)
0926 13:18:58.874118 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.874143 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:18:58.874330 22689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33411: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33411: connect: Connection refused (error 111)
I20260926 13:18:58.884807 22696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:58.887204 22696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:58.887377 22696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:58.891283 22696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33411
I20260926 13:18:58.891331 22763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33411 every 8 connection(s)
I20260926 13:18:58.891675 22696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:18:58.892666 22764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:58.894220 22764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:58.898404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22696
I20260926 13:18:58.898515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-1/wal/instance
I20260926 13:18:58.898730 392 external_mini_cluster.cc:1428] Setting key 0b55421dcf97ed593211f3e157a8c2d2
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:18:58.903622 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.894940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.0.98.62:33087)
0926 13:18:58.895006 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:58.895008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:58.895230 (+ 222us) server_negotiation.cc:407] Connection header received
0926 13:18:58.895741 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:58.895742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:58.895758 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:58.895810 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:58.896432 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.896963 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.897961 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.898217 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.898278 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.900682 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:58.900684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:58.900684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:58.900692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:58.902230 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.902640 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.902641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.902641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.902701 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.902950 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.902951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.902951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.903072 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:58.903148 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:58.903435 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 13:18:58.903521 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:18:58.903816 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.894915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33087 (local address 127.0.0.1:38504)
0926 13:18:58.895110 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.895145 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:58.895625 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:58.895884 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:58.895888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:58.896142 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:58.896333 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.896337 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.897062 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.897067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.897869 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.897871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.898339 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.898341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.898458 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.899073 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:58.899082 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:58.900582 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:58.902316 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.902323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.902325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.902555 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.902766 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.902766 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.902767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.902879 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.903139 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:58.903145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:58.903300 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 13:18:58.903416 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 13:18:58.903578 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260926 13:18:58.904392 22764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:18:58.912225 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.904679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41604)
0926 13:18:58.904931 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.904943 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:58.905005 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:58.905168 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:58.905169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:58.905320 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:58.905384 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.905387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.906311 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.906312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.906912 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.906915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.907392 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.907392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.907445 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.907818 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:58.907822 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:58.909431 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:58.911315 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.911315 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.911316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.911461 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.911686 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.911686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.911686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.911727 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.912010 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:58.912011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:58.912043 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 13:18:58.912092 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 13:18:58.912162 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260926 13:18:58.912366 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.904771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.0.98.61:33411)
0926 13:18:58.904937 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:58.904940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:58.904996 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:18:58.905064 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:58.905066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:58.905077 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:58.905135 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:58.905464 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.906235 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.906995 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.907326 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.907383 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.909511 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:58.909518 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:58.909518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:58.909524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:58.911245 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.911538 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.911541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.911541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.911623 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.911764 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.911764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.911765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.911946 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:58.912027 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:58.912150 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 13:18:58.912239 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
I20260926 13:18:58.913005 22764 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:58.913352 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.913252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50678)
0926 13:18:58.913293 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.913334 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:18:58.913520 22714 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
I20260926 13:18:58.913954 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.913870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50686)
0926 13:18:58.913914 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.913938 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 13:18:58.914067 22764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 13:18:58.925329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35075
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33087,127.0.98.61:33411,127.0.98.60:35075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:18:58.963363 22764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } attempt: 1
I20260926 13:18:58.963924 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.963772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50694)
0926 13:18:58.963839 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.963888 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 13:18:58.964361 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.964219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50704)
0926 13:18:58.964297 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.964332 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 13:18:58.964526 22764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
I20260926 13:18:58.965477 22689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } attempt: 3
Sep 26 13:18:58 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:18:58.973361 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.965850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.0.98.61:33411)
0926 13:18:58.965903 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:58.965906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:58.965921 (+ 15us) server_negotiation.cc:407] Connection header received
0926 13:18:58.966026 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:58.966028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:58.966040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:58.966135 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:58.966506 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.967092 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.967915 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.968163 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.968245 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.970453 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:58.970454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:58.970455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:58.970464 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:58.972080 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.972491 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.972492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.972493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.972556 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:58.972776 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:58.972777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:58.972777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:58.972914 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:58.973000 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:58.973168 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 13:18:58.973252 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 13:18:58.973362 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.965802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41616)
0926 13:18:58.965856 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.965876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:58.965937 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:58.966144 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:58.966145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:58.966341 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:58.966414 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.966416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.967211 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.967212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.967813 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:58.967814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:58.968256 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:58.968258 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:58.968342 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:58.968862 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:58.968867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:58.970349 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:58.972194 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.972195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.972195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.972387 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.972631 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:58.972632 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:58.972633 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:58.972697 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:58.972990 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:58.972991 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:58.973043 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 13:18:58.973141 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 13:18:58.973252 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 13:18:58.973958 22689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:58.974354 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.974244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50716)
0926 13:18:58.974298 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.974331 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 13:18:58.974774 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:58.974648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50720)
0926 13:18:58.974726 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:58.974754 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 13:18:58.974903 22689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
I20260926 13:18:59.006053 22689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } attempt: 1
I20260926 13:18:59.006538 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.006400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50730)
0926 13:18:59.006468 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.006509 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 13:18:59.007014 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.006918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50744)
0926 13:18:59.006969 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.006995 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 13:18:59.007162 22689 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
W20260926 13:18:59.011067 22771 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:18:59.011288 22771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:18:59.011310 22771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:18:59.011334 22771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:18:59.012547 22771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:18:59.012589 22771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:18:59.012606 22771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:18:59.012622 22771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:18:59.012637 22771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:18:59.014611 22771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33087,127.0.98.61:33411,127.0.98.60:35075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:18:59.014814 22771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:18:59.014990 22771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:18:59.020040 22776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:18:59.019946 22777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:59.020110 22771 server_base.cc:1034] running on GCE node
W20260926 13:18:59.019994 22779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:18:59.020443 22771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:18:59.020782 22771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:18:59.021937 22771 hybrid_clock.cc:648] HybridClock initialized: now 1790428739021927 us; error 44 us; skew 500 ppm
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:18:59.024397 22771 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 13:18:59.024829 22771 webserver.cc:460] Webserver started at http://127.0.98.60:41411/ using document root <none> and password file <none>
I20260926 13:18:59.025053 22771 fs_manager.cc:359] Metadata directory not provided
I20260926 13:18:59.025120 22771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:18:59.025269 22771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:18:59.026382 22771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data/instance:
uuid: "51d671d3e76c47f5824764f64aa62aa9"
format_stamp: "Formatted at 2026-09-26 13:18:59 on dist-test-slave-f864"
server_key: "7734b3890a4fb9708185e2122f595408"
server_key_iv: "4ddce5a986422782251becd8533ba6f9"
server_key_version: "encryptionkey@0"
I20260926 13:18:59.026743 22771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal/instance:
uuid: "51d671d3e76c47f5824764f64aa62aa9"
format_stamp: "Formatted at 2026-09-26 13:18:59 on dist-test-slave-f864"
server_key: "7734b3890a4fb9708185e2122f595408"
server_key_iv: "4ddce5a986422782251becd8533ba6f9"
server_key_version: "encryptionkey@0"
I20260926 13:18:59.028031 22771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:59.028820 22786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:18:59.029006 22771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:18:59.029104 22771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal
uuid: "51d671d3e76c47f5824764f64aa62aa9"
format_stamp: "Formatted at 2026-09-26 13:18:59 on dist-test-slave-f864"
server_key: "7734b3890a4fb9708185e2122f595408"
server_key_iv: "4ddce5a986422782251becd8533ba6f9"
server_key_version: "encryptionkey@0"
I20260926 13:18:59.029188 22771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:18:59.032686 22764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } attempt: 2
I20260926 13:18:59.033093 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.032950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50752)
0926 13:18:59.033012 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.033058 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:18:59.033412 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.033329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50766)
0926 13:18:59.033369 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.033394 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:18:59.033530 22764 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35075: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35075: connect: Connection refused (error 111)
I20260926 13:18:59.060607 22771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:18:59.063067 22771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:18:59.063235 22771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:18:59.067096 22771 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35075
I20260926 13:18:59.067173 22838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35075 every 8 connection(s)
I20260926 13:18:59.067510 22771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:18:59.068468 22839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:18:59.069988 22839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:59.070371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 22771
I20260926 13:18:59.070464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790428444292953-392-0/minicluster-data/master-2/wal/instance
I20260926 13:18:59.070639 392 external_mini_cluster.cc:1428] Setting key 5d1e99a32065935aabafc83805737e22
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:18:59.080365 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.070776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.0.98.62:33087)
0926 13:18:59.070829 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.070831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.071102 (+ 271us) server_negotiation.cc:407] Connection header received
0926 13:18:59.071677 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.071678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.071690 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.071744 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.072468 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.073239 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.074408 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.074641 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.074715 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.077039 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.077041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.077041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.077049 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.079137 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.079519 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.079520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.079520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.079576 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.079797 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.079798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.079798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.079909 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.079983 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.080189 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.080265 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:18:59.080463 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.070745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33087 (local address 127.0.0.1:38510)
0926 13:18:59.070903 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.070969 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:59.071581 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:59.071772 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:59.071775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:59.072135 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:59.072369 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.072374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.073318 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.073319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.074327 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.074329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.074735 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.074736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.074830 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.075337 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:59.075345 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:59.076949 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:59.079242 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.079249 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.079251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.079442 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.079669 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.079670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.079670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.079734 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.079976 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:59.079982 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:59.080048 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 13:18:59.080159 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 13:18:59.080265 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260926 13:18:59.080471 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.071001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.0.98.60:35075)
0926 13:18:59.071156 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.071159 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.071324 (+ 165us) server_negotiation.cc:407] Connection header received
0926 13:18:59.071580 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.071586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.071610 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.071724 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.072471 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.073569 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.074554 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.074928 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.074979 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.077489 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.077497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.077497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.077504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.078961 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.079309 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.079312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.079313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.079392 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.079607 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.079608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.079608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.079820 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.079966 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.080067 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.080190 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260926 13:18:59.080941 22839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:59.082296 22689 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } attempt: 2
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:18:59.089349 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.081276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.0.98.61:33411)
0926 13:18:59.081320 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.081322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.081396 (+ 74us) server_negotiation.cc:407] Connection header received
0926 13:18:59.081425 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.081426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.081435 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.081499 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.081692 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.082309 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.083033 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.083328 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.083393 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.085111 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.085112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.085112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.085120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.087280 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.088214 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.088215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.088215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.088276 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.088602 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.088603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.088603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.088745 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.088811 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.089166 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.089247 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:18:59.089659 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.081221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33411 (local address 127.0.0.1:41630)
0926 13:18:59.081311 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.081320 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:59.081360 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:59.081503 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:59.081503 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:59.081615 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:59.081642 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.081643 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.082389 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.082390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.082959 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.082961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.083404 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.083405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.083464 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.083873 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:59.083877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:59.085023 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:59.087362 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.087363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.087363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.088132 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.088476 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.088477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.088477 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.088529 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.088966 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:59.088967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:59.089012 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 13:18:59.089451 (+ 439us) client_negotiation.cc:240] Negotiation successful
0926 13:18:59.089547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 13:18:59.089859 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.081428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.0.98.62:33087)
0926 13:18:59.081468 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.081470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.081600 (+ 130us) server_negotiation.cc:407] Connection header received
0926 13:18:59.081667 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.081668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.081677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.081720 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.082150 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.082882 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.083671 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.083962 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.084035 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.086579 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.086580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.086580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.086587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.088330 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.088691 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.088692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.088693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.088740 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.088944 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.088944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.088945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.089033 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.089219 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.089256 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.089736 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":20}
I20260926 13:18:59.090019 22839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:18:59.090298 22691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.082515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50780)
0926 13:18:59.082570 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.082581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:59.082617 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:59.082803 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:59.082804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:59.082959 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:59.082989 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.082990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.083852 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.083853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.084652 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.084655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.085285 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.085287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.085341 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.085755 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:59.085761 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:59.087310 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:59.089212 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.089213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.089213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.089454 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.089646 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.089647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.089647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.089710 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.089984 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:59.089985 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:59.090028 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:18:59.090079 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 13:18:59.090201 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":4}
I20260926 13:18:59.090427 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.082679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.0.98.60:35075)
0926 13:18:59.082709 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.082711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.082717 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:18:59.082732 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.082733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.082740 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.083065 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.083089 (+ 24us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.083760 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.084753 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.085192 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.085478 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.087392 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.087393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.087394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.087399 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.089130 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.089529 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.089530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.089530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.089579 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.089778 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.089779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.089779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.089896 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.090231 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.090280 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.090350 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 13:18:59.091856 22689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2: Bootstrap starting.
I20260926 13:18:59.092574 22689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:59.092916 22689 log.cc:824] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:59.094012 22689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2: No bootstrap required, opened a new log
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:18:59.096298 22689 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.096472 22689 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9cb9b6177e7426bb99625e728c77ae2, State: Initialized, Role: FOLLOWER
I20260926 13:18:59.096575 22689 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.096870 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.096937 22689 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:18:59.098074 22841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.090296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50794)
0926 13:18:59.090834 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.090845 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:59.090887 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:59.090998 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:59.090999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:59.091138 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:59.091165 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.091167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.091751 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.091752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.092423 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.092426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.092829 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.092830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.092894 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.093300 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:59.093304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:59.094670 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:59.097070 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.097071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.097071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.097223 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.097434 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.097435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.097436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.097492 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.097765 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:59.097765 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:59.097802 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 13:18:59.097885 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 13:18:59.097975 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523}
I20260926 13:18:59.098206 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.090363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.60:35075)
0926 13:18:59.090622 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.090624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.090877 (+ 253us) server_negotiation.cc:407] Connection header received
0926 13:18:59.090936 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.090936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.090944 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.090994 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.091234 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.091671 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.092499 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.092759 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.092812 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.094754 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.094754 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.094754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.094760 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.096954 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.097310 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.097311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.097311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.097361 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.097541 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.097542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.097542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.097711 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.097771 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.097918 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.098035 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":239,"thread_start_us":88,"threads_started":1}
I20260926 13:18:59.099404 22839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: Bootstrap starting.
W20260926 13:18:59.099709 22857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:18:59.099758 22857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:18:59.099928 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:18:59.100095 22839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:59.100418 22839 log.cc:824] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:59.101485 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.090495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.0.98.61:33411)
0926 13:18:59.090535 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.090537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.090646 (+ 109us) server_negotiation.cc:407] Connection header received
0926 13:18:59.090745 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.090746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.090757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.090814 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.094026 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.094630 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.095417 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.095630 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.095702 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.098486 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.098487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.098487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.098494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.100303 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.100691 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.100692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.100692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.100750 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.101011 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.101012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.101012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.101128 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.101285 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.101324 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.101389 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:18:59.103199 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 13:18:59.100857 22621 master_runner.cc:186] Error getting master registration for 127.0.98.61:33411: OK, instance_id { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" instance_seqno: 1790428738890077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:18:59.103790 22621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:18:59.103994 22663 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d9cb9b6177e7426bb99625e728c77ae2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38516
I20260926 13:18:59.104105 22663 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:18:59.104147 22663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:59.104310 22839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: No bootstrap required, opened a new log
I20260926 13:18:59.105000 22663 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.105209 22663 leader_election.cc:288] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [CANDIDATE]: Term 1 election: Requested vote from peers 05c6e8c521464c14adc5a7efc6ed19e2 (127.0.98.61:33411), 51d671d3e76c47f5824764f64aa62aa9 (127.0.98.60:35075)
I20260926 13:18:59.106884 22839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.107066 22839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d671d3e76c47f5824764f64aa62aa9, State: Initialized, Role: FOLLOWER
I20260926 13:18:59.107165 22839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.107457 22839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:59.107528 22813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9cb9b6177e7426bb99625e728c77ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51d671d3e76c47f5824764f64aa62aa9"
I20260926 13:18:59.107640 22813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:59.108512 22813 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cb9b6177e7426bb99625e728c77ae2 in term 1.
I20260926 13:18:59.108812 22639 leader_election.cc:302] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51d671d3e76c47f5824764f64aa62aa9, d9cb9b6177e7426bb99625e728c77ae2; no voters:
I20260926 13:18:59.108935 22846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:18:59.109091 22846 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [term 1 LEADER]: Becoming Leader. State: Replica: d9cb9b6177e7426bb99625e728c77ae2, State: Running, Role: LEADER
I20260926 13:18:59.109181 22846 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.109596 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9cb9b6177e7426bb99625e728c77ae2. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.109704 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:59.110462 22881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 13:18:59.110574 22878 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:18:59.110646 22878 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:18:59.110890 22881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:18:59.112157 22771 master_runner.cc:186] Error getting master registration for 127.0.98.61:33411: OK, instance_id { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" instance_seqno: 1790428738890077 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:18:59.114365 22771 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:18:59.115459 22813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [term 1 FOLLOWER]: Refusing update from remote peer d9cb9b6177e7426bb99625e728c77ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:59.115806 22846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:59.117516 22692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.106309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.0.98.62:33087)
0926 13:18:59.106349 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.106351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.106997 (+ 646us) server_negotiation.cc:407] Connection header received
0926 13:18:59.107296 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.107297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.107310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.107349 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.109791 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.110486 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.111857 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.112143 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.112184 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.113736 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.113737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.113738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.113745 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.116284 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.116695 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.116696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.116696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.116748 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.116944 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.116944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.116945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.117057 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.117142 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.117358 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.117428 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 13:18:59.123767 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.106487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50810 (local address 127.0.98.60:35075)
0926 13:18:59.106528 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.106530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.114298 (+ 7768us) server_negotiation.cc:407] Connection header received
0926 13:18:59.114398 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.114399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.114410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.114441 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.115072 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.115767 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.116715 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.117018 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.117063 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.119988 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.119989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.119989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.119996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.121766 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.122919 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.122921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.122921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.122977 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.123182 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.123183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.123183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.123296 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.123400 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.123604 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.123685 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 13:18:59.124672 22764 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } attempt: 3
I20260926 13:18:59.125495 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.125603 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:18:59.125660 22884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9cb9b6177e7426bb99625e728c77ae2. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.125705 22884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 13:18:59 dist-test-slave-f864 krb5kdc[22608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:18:59.134327 22766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.124929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35075 (local address 127.0.0.1:50826)
0926 13:18:59.124970 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:18:59.124981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 13:18:59.125051 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:18:59.125178 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:18:59.125179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:18:59.125341 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:18:59.125383 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.125385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.126429 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.126430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.127233 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:18:59.127235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.128231 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.128232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:18:59.128313 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.128919 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:18:59.128924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:18:59.130997 (+ 2073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:18:59.133321 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.133322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.133322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.133515 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.133721 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:18:59.133721 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:18:59.133722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:18:59.133774 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:18:59.134021 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:18:59.134022 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:18:59.134061 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 13:18:59.134122 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 13:18:59.134228 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 13:18:59.134666 22844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.124964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.0.98.60:35075)
0926 13:18:59.124997 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.124998 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.125054 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:18:59.125110 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.125111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.125121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.125175 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.125486 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.126348 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.127358 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.128063 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.128526 (+ 463us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.131089 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.131090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.131090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.131097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.133233 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.133602 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.133602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.133603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.133653 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.133835 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.133835 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.133836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.133945 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.133999 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.134499 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.134584 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 13:18:59.135932 22768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:18:59.106771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.98.61:33411)
0926 13:18:59.106813 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.106815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.117585 (+ 10770us) server_negotiation.cc:407] Connection header received
0926 13:18:59.117694 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.117694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.117706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.117753 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.118898 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.119592 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.122621 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:18:59.122945 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:18:59.123010 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:18:59.126723 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:18:59.126724 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:18:59.126724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:18:59.126731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:18:59.132474 (+ 5743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.135142 (+ 2668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.135143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.135143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.135194 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:18:59.135374 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:18:59.135375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:18:59.135375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:18:59.135484 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:18:59.135582 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 13:18:59.135776 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 13:18:59.135848 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:18:59.136188 22764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: Bootstrap starting.
I20260926 13:18:59.136883 22764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: Neither blocks nor log segments found. Creating new log.
I20260926 13:18:59.137151 22764 log.cc:824] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: Log is configured to *not* fsync() on all Append() calls
I20260926 13:18:59.138404 22764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: No bootstrap required, opened a new log
I20260926 13:18:59.138463 22879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.138569 22879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:59.139104 22867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.139218 22867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:18:59.139292 22885 mvcc.cc:205] Tried to move back new op lower bound from 7333596115413757952 to 7333596115392012288. Current Snapshot: MvccSnapshot[applied={T|T < 7333596115413757952}]
I20260926 13:18:59.139415 22881 catalog_manager.cc:1347] Generated new cluster ID: 4892946c58894e8381779afc5bca67a8
I20260926 13:18:59.139469 22881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:18:59.141043 22764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.141167 22764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05c6e8c521464c14adc5a7efc6ed19e2, State: Initialized, Role: FOLLOWER
I20260926 13:18:59.141239 22764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } }
I20260926 13:18:59.141526 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.141707 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:18:59.141564 22738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9cb9b6177e7426bb99625e728c77ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "05c6e8c521464c14adc5a7efc6ed19e2"
I20260926 13:18:59.141618 22737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:18:59.142804 22737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [term 1 FOLLOWER]: Refusing update from remote peer d9cb9b6177e7426bb99625e728c77ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:18:59.143092 22846 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:18:59.143963 22738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9cb9b6177e7426bb99625e728c77ae2 in term 1.
I20260926 13:18:59.141547 22764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:18:59.147120 22893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.147265 22893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:18:59.146976 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9cb9b6177e7426bb99625e728c77ae2. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.148021 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:18:59.148165 22905 catalog_manager.cc:1259] Loaded cluster ID: 4892946c58894e8381779afc5bca67a8
I20260926 13:18:59.148205 22905 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: loading cluster ID for follower catalog manager: success
W20260926 13:18:59.148281 22905 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260926 13:18:59.148308 22905 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:18:59.161350 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9cb9b6177e7426bb99625e728c77ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9cb9b6177e7426bb99625e728c77ae2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33087 } } peers { permanent_uuid: "05c6e8c521464c14adc5a7efc6ed19e2" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33411 } } peers { permanent_uuid: "51d671d3e76c47f5824764f64aa62aa9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35075 } } }
I20260926 13:18:59.161458 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2 [sys.catalog]: This master's current role is: LEADER
I20260926 13:18:59.165898 22881 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:18:59.166707 22881 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:18:59.178885 22881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9cb9b6177e7426bb99625e728c77ae2: Generated new TSK 0
I20260926 13:18:59.179203 22881 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 13:18:59.207666 22844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.206061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50832 (local address 127.0.98.60:35075)
0926 13:18:59.206104 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.206107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.206983 (+ 876us) server_negotiation.cc:407] Connection header received
0926 13:18:59.207088 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.207089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.207100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.207130 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.207595 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50832: BlockingRecv error: recv got EOF from 127.0.0.1:50832 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:18:59.208628 22692 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.206438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.0.98.62:33087)
0926 13:18:59.206476 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.206478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.208067 (+ 1589us) server_negotiation.cc:407] Connection header received
0926 13:18:59.208208 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.208209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.208219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.208247 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.208569 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38538: BlockingRecv error: recv got EOF from 127.0.0.1:38538 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:18:59.209435 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.206752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.0.98.61:33411)
0926 13:18:59.206788 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.206791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.208930 (+ 2139us) server_negotiation.cc:407] Connection header received
0926 13:18:59.209020 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.209021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.209031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.209061 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.209376 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41654: BlockingRecv error: recv got EOF from 127.0.0.1:41654 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:18:59.211740 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.211037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.0.98.61:33411)
0926 13:18:59.211071 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.211072 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.211265 (+ 193us) server_negotiation.cc:407] Connection header received
0926 13:18:59.211366 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.211368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.211385 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.211411 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.211699 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41670: BlockingRecv error: recv got EOF from 127.0.0.1:41670 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:18:59.214778 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.214056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.0.98.61:33411)
0926 13:18:59.214088 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.214090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.214320 (+ 230us) server_negotiation.cc:407] Connection header received
0926 13:18:59.214409 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.214410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.214420 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.214448 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.214738 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41674: BlockingRecv error: recv got EOF from 127.0.0.1:41674 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:18:59.218775 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.218002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.0.98.61:33411)
0926 13:18:59.218035 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.218037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.218279 (+ 242us) server_negotiation.cc:407] Connection header received
0926 13:18:59.218375 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.218376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.218386 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.218416 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.218735 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41684: BlockingRecv error: recv got EOF from 127.0.0.1:41684 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:18:59.224710 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.223940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41696 (local address 127.0.98.61:33411)
0926 13:18:59.223974 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.223976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.224185 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:18:59.224280 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.224281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.224290 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.224317 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.224663 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41696: BlockingRecv error: recv got EOF from 127.0.0.1:41696 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 13:18:59.234948 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.234045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.98.61:33411)
0926 13:18:59.234083 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.234085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.234364 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:18:59.234469 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.234470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.234481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.234511 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.234891 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41710: BlockingRecv error: recv got EOF from 127.0.0.1:41710 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 13:18:59.253513 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.252567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.0.98.61:33411)
0926 13:18:59.252616 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.252619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.252883 (+ 264us) server_negotiation.cc:407] Connection header received
0926 13:18:59.252995 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.252996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.253009 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.253040 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.253446 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41718: BlockingRecv error: recv got EOF from 127.0.0.1:41718 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 13:18:59.287989 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.287103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.0.98.61:33411)
0926 13:18:59.287148 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.287150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.287395 (+ 245us) server_negotiation.cc:407] Connection header received
0926 13:18:59.287493 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.287494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.287506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.287548 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.287929 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41734: BlockingRecv error: recv got EOF from 127.0.0.1:41734 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 13:18:59.354570 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.353739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.0.98.61:33411)
0926 13:18:59.353806 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.353808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.353937 (+ 129us) server_negotiation.cc:407] Connection header received
0926 13:18:59.353987 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.353988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.353997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.354049 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.354486 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41750: BlockingRecv error: recv got EOF from 127.0.0.1:41750 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 13:18:59.484541 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.483812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41754 (local address 127.0.98.61:33411)
0926 13:18:59.483884 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.483886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.483992 (+ 106us) server_negotiation.cc:407] Connection header received
0926 13:18:59.484045 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.484046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.484057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.484113 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.484471 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41754: BlockingRecv error: recv got EOF from 127.0.0.1:41754 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 13:18:59.742772 22768 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:18:59.741943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41766 (local address 127.0.98.61:33411)
0926 13:18:59.742014 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 13:18:59.742016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:18:59.742087 (+ 71us) server_negotiation.cc:407] Connection header received
0926 13:18:59.742137 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:18:59.742138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:18:59.742148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:18:59.742260 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:18:59.742705 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41766: BlockingRecv error: recv got EOF from 127.0.0.1:41766 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 13:19:00.111068 22878 catalog_manager.cc:1259] Loaded cluster ID: 4892946c58894e8381779afc5bca67a8
I20260926 13:19:00.111155 22878 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: loading cluster ID for follower catalog manager: success
I20260926 13:19:00.111912 22878 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: acquiring CA information for follower catalog manager: success
I20260926 13:19:00.112162 22878 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51d671d3e76c47f5824764f64aa62aa9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:19:00.149355 22905 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: acquiring CA information for follower catalog manager: success
I20260926 13:19:00.149739 22905 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 05c6e8c521464c14adc5a7efc6ed19e2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:19:00.258744 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:00.255907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.0.98.61:33411)
0926 13:19:00.256048 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:00.256051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:00.256107 (+ 56us) server_negotiation.cc:407] Connection header received
0926 13:19:00.256214 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:00.256215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:00.256285 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:00.256377 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:00.256557 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.257063 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.257851 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.258073 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.258154 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:00.258351 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:00.258511 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:00.258595 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:00.258617 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 13:19:00.258640 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":54,"threads_started":1}
I20260926 13:19:00.263250 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:00.260607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.0.98.60:35075)
0926 13:19:00.260742 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:00.260745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:00.260795 (+ 50us) server_negotiation.cc:407] Connection header received
0926 13:19:00.260880 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:00.260881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:00.260928 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:00.261026 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:00.261120 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.261598 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.262444 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.262652 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.262727 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:00.262892 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:00.263005 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:00.263100 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:00.263126 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 13:19:00.263149 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":48,"threads_started":1}
I20260926 13:19:00.268275 22986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:00.264972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.0.98.62:33087)
0926 13:19:00.265281 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:00.265283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:00.265289 (+ 6us) server_negotiation.cc:407] Connection header received
0926 13:19:00.265357 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:00.265358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:00.265406 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:00.265497 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:00.265581 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.266356 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.267405 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.267646 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.267720 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:00.267897 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:00.267979 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:00.268093 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:00.268162 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 13:19:00.268186 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":56,"threads_started":1}
I20260926 13:19:00.268304 22973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:00.265067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.0.98.61:33411)
0926 13:19:00.265104 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:00.265106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:00.265114 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:19:00.265139 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:00.265139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:00.265150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:00.265217 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:00.265385 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.266187 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.266982 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.267188 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.267269 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:00.268039 (+ 770us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:00.268074 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:00.268124 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:00.268220 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 13:19:00.268239 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:19:00.268955 22979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:00.264949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50846 (local address 127.0.98.60:35075)
0926 13:19:00.265007 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:00.265009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:00.265338 (+ 329us) server_negotiation.cc:407] Connection header received
0926 13:19:00.266275 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:00.266276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:00.266286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:00.266340 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:00.266472 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.267176 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.267892 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:00.268308 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:00.268389 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:00.268693 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:00.268723 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:00.268782 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:00.268868 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 13:19:00.268886 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 13:19:00.270272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22621
I20260926 13:19:00.276104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22696
I20260926 13:19:00.280681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 22771
2026-09-26T13:19:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2803 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 13:19:00 dist-test-slave-f864 krb5kdc[22990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 13:19:00 dist-test-slave-f864 krb5kdc[22990](info): set up 2 sockets
Sep 26 13:19:00 dist-test-slave-f864 krb5kdc[22990](info): commencing operation
krb5kdc: starting...
W20260926 13:19:01.318894 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/test-user.keytab.
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T13:19:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T13:19:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 13:19:01.438298 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38923
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38923,127.0.98.61:43193,127.0.98.60:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 13:19:01.519511 23003 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:19:01.519731 23003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:19:01.519748 23003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:19:01.519769 23003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:19:01.520862 23003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:19:01.520897 23003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:19:01.520912 23003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:19:01.520931 23003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:19:01.520944 23003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:19:01.522428 23003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38923,127.0.98.61:43193,127.0.98.60:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:19:01.522620 23003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:19:01.522907 23003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:19:01.527781 23011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:19:01.527822 23008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:19:01.527829 23009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:19:01.528072 23003 server_base.cc:1034] running on GCE node
I20260926 13:19:01.528414 23003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:19:01.528744 23003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:19:01.529940 23003 hybrid_clock.cc:648] HybridClock initialized: now 1790428741529926 us; error 29 us; skew 500 ppm
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:19:01.549345 23003 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 13:19:01.549839 23003 webserver.cc:460] Webserver started at http://127.0.98.62:36475/ using document root <none> and password file <none>
I20260926 13:19:01.550105 23003 fs_manager.cc:359] Metadata directory not provided
I20260926 13:19:01.550207 23003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:19:01.550379 23003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:19:01.551509 23003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data/instance:
uuid: "17990373fe2c49b8bb3aee6b20bcd478"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "dd05b076c16c7a725cbe026e17a47734"
server_key_iv: "31e9174e45f17b06a1adf6ceb8819ba5"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.551888 23003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal/instance:
uuid: "17990373fe2c49b8bb3aee6b20bcd478"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "dd05b076c16c7a725cbe026e17a47734"
server_key_iv: "31e9174e45f17b06a1adf6ceb8819ba5"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.553298 23003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 13:19:01.554229 23018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:19:01.554430 23003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:19:01.554551 23003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal
uuid: "17990373fe2c49b8bb3aee6b20bcd478"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "dd05b076c16c7a725cbe026e17a47734"
server_key_iv: "31e9174e45f17b06a1adf6ceb8819ba5"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.554651 23003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:19:01.567121 23003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:19:01.569252 23003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:19:01.569450 23003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:19:01.574419 23003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38923
I20260926 13:19:01.574491 23070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38923 every 8 connection(s)
I20260926 13:19:01.574942 23003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/data/info.pb
I20260926 13:19:01.575935 23071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:19:01.577440 23071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:19:01.583760 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 23003
I20260926 13:19:01.583909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-0/wal/instance
I20260926 13:19:01.584188 392 external_mini_cluster.cc:1428] Setting key f72f9a5ceb465058769428443d8e5d1e
I20260926 13:19:01.588156 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.578094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.0.1:51110)
0926 13:19:01.578342 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.578415 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.578925 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.579211 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.579214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.579504 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.579665 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.579669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.580707 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.580708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.581397 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.581399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.581895 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.581897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.581958 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.582500 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.582509 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.583882 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.586299 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.586307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.586309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.586664 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.586969 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.586970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.586971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.587103 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.587557 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.587562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.587636 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.587740 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.587983 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":96,"thread_start_us":173,"threads_started":1}
I20260926 13:19:01.588160 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.578144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.98.62:38923)
0926 13:19:01.578306 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.578317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.578518 (+ 201us) server_negotiation.cc:407] Connection header received
0926 13:19:01.579097 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.579099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.579115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.579208 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.579762 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.580594 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.581494 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.581792 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.581917 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.584005 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.584014 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.584015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.584024 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.586151 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.586771 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.586774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.586774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.586876 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.587197 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.587198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.587199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.587417 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.587578 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.587764 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.587912 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260926 13:19:01.588936 23071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:19:01.589330 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.589217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34938)
0926 13:19:01.589267 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.589310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 13:19:01.589531 23020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
I20260926 13:19:01.590003 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.589912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34942)
0926 13:19:01.589961 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.589986 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 13:19:01.590148 23071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 13:19:01.612717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43193
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38923,127.0.98.61:43193,127.0.98.60:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:19:01.639434 23071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } attempt: 1
I20260926 13:19:01.640061 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.639886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34956)
0926 13:19:01.639958 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.640016 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 13:19:01.640574 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.640460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34966)
0926 13:19:01.640517 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.640549 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 13:19:01.640691 23071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
W20260926 13:19:01.694324 23078 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:19:01.694542 23078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:19:01.694604 23078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:19:01.694648 23078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:19:01.695953 23078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:19:01.696009 23078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:19:01.696025 23078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:19:01.696038 23078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:19:01.696053 23078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:19:01.697611 23078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38923,127.0.98.61:43193,127.0.98.60:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:19:01.697804 23078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:19:01.697993 23078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:19:01.702925 23084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:19:01.702921 23083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:19:01.702958 23086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:19:01.703174 23078 server_base.cc:1034] running on GCE node
I20260926 13:19:01.703436 23078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:19:01.703727 23078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:19:01.704895 23078 hybrid_clock.cc:648] HybridClock initialized: now 1790428741704893 us; error 39 us; skew 500 ppm
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:19:01.707594 23078 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 13:19:01.708207 23078 webserver.cc:460] Webserver started at http://127.0.98.61:42331/ using document root <none> and password file <none>
I20260926 13:19:01.708506 23078 fs_manager.cc:359] Metadata directory not provided
I20260926 13:19:01.708595 23078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:19:01.708739 23078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:19:01.708834 23071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } attempt: 2
I20260926 13:19:01.709254 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.709124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34970)
0926 13:19:01.709187 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.709224 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:19:01.709579 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.709467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34984)
0926 13:19:01.709522 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.709547 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 13:19:01.709669 23071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43193: connect: Connection refused (error 111)
I20260926 13:19:01.710047 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data/instance:
uuid: "93d819f0d9c14a0d92ea09aa2f4b242f"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "565b332ef7f8bc6d632452f1d0cc478e"
server_key_iv: "56e31c160e7b9eb079aad31ad5376b23"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.710479 23078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal/instance:
uuid: "93d819f0d9c14a0d92ea09aa2f4b242f"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "565b332ef7f8bc6d632452f1d0cc478e"
server_key_iv: "56e31c160e7b9eb079aad31ad5376b23"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.711807 23078 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:19:01.712563 23093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:19:01.712745 23078 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:19:01.712857 23078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal
uuid: "93d819f0d9c14a0d92ea09aa2f4b242f"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "565b332ef7f8bc6d632452f1d0cc478e"
server_key_iv: "56e31c160e7b9eb079aad31ad5376b23"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.712937 23078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:19:01.743992 23078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:19:01.746577 23078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:19:01.746788 23078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:19:01.751099 23078 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43193
I20260926 13:19:01.751130 23145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43193 every 8 connection(s)
I20260926 13:19:01.751526 23078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/data/info.pb
I20260926 13:19:01.752600 23146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:19:01.754335 23146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:19:01.758006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 23078
I20260926 13:19:01.758103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-1/wal/instance
I20260926 13:19:01.758327 392 external_mini_cluster.cc:1428] Setting key 7c711904ddd29647490e78dbfae66da4
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:19:01.764424 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.755179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.0.98.62:38923)
0926 13:19:01.755274 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.755276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.755479 (+ 203us) server_negotiation.cc:407] Connection header received
0926 13:19:01.755970 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.755971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.755982 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.756043 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.756691 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.757279 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.758648 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.758926 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.759063 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.761385 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.761386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.761386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.761396 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.762993 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.763426 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.763428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.763428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.763502 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.763758 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.763759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.763760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.763880 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.763953 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.764243 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.764331 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 13:19:01.764632 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.755118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.0.1:51114)
0926 13:19:01.755320 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.755372 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.755869 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.756120 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.756123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.756375 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.756576 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.756580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.757353 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.757354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.758540 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.758542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.759089 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.759091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.759238 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.759955 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.759963 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.761281 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.763094 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.763101 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.763104 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.763334 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.763571 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.763572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.763573 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.763687 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.763964 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.763971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.764110 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.764221 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.764376 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":116,"threads_started":1}
I20260926 13:19:01.765198 23146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:19:01.772614 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.765471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:34992)
0926 13:19:01.765520 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.765700 (+ 180us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.765744 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.765885 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.765886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.766024 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.766064 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.766066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.766950 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.766951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.767462 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.767463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.767958 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.767959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.768021 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.768451 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.768461 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.769697 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.771544 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.771544 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.771545 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.771703 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.771918 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.771919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.771920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.771972 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.772261 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.772262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.772299 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.772372 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.772479 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 13:19:01.772782 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.765558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.0.98.61:43193)
0926 13:19:01.765719 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.765723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.765744 (+ 21us) server_negotiation.cc:407] Connection header received
0926 13:19:01.765802 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.765805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.765817 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.765877 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.766137 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.766854 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.767557 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.767872 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.767966 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.769779 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.769787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.769787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.769794 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.771442 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.771794 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.771797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.771797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.771848 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.772052 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.772053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.772053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.772192 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.772283 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.772400 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.772531 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260926 13:19:01.773399 23146 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:19:01.773747 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.773629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52490)
0926 13:19:01.773674 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.773717 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 13:19:01.773916 23094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
I20260926 13:19:01.774417 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.774326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52496)
0926 13:19:01.774376 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.774400 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 13:19:01.774551 23146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 13:19:01.786015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskLNUYBk/build/release/bin/kudu
/tmp/dist-test-taskLNUYBk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37447
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38923,127.0.98.61:43193,127.0.98.60:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 13:19:01.800817 23071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } attempt: 3
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:19:01.808184 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.801186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:35002)
0926 13:19:01.801250 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.801263 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.801338 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.801516 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.801517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.801684 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.801753 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.801755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.802505 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.802506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.803023 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.803025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.803569 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.803570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.803666 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.804126 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.804132 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.805554 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.807266 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.807267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.807267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.807435 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.807635 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.807635 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.807636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.807686 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.807940 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.807940 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.807983 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.808033 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.808111 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 13:19:01.808244 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.801263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35002 (local address 127.0.98.61:43193)
0926 13:19:01.801331 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.801333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.801345 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:19:01.801397 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.801399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.801410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.801505 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.801843 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.802407 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.803113 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.803399 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.803526 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.805643 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.805645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.805645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.805653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.807173 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.807511 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.807512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.807512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.807571 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.807745 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.807746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.807746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.807881 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.807945 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.808077 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.808157 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 13:19:01.808636 23071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:19:01.808966 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.808868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52498)
0926 13:19:01.808910 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.808941 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 13:19:01.809254 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.809199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52510)
0926 13:19:01.809220 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.809240 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 13:19:01.809362 23071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
I20260926 13:19:01.823757 23146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } attempt: 1
I20260926 13:19:01.824221 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.824099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52518)
0926 13:19:01.824158 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.824194 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 13:19:01.824659 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.824574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52534)
0926 13:19:01.824619 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.824643 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 13:19:01.824769 23146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
I20260926 13:19:01.840566 23071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } attempt: 1
I20260926 13:19:01.841071 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.840922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52544)
0926 13:19:01.840986 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.841037 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:19:01.841408 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.841330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52546)
0926 13:19:01.841362 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.841389 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 13:19:01.841533 23071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
W20260926 13:19:01.871358 23153 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 13:19:01.871577 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 13:19:01.871594 23153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 13:19:01.871614 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 13:19:01.872819 23153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 13:19:01.872870 23153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 13:19:01.872884 23153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 13:19:01.872934 23153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 13:19:01.872969 23153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 13:19:01.874560 23153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38923,127.0.98.61:43193,127.0.98.60:37447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 12:15:03 UTC on ccycloud.quasar-vyifkg.root.comops.site
build id 4674112
I20260926 13:19:01.874821 23153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 13:19:01.875080 23153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 13:19:01.880050 23161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:19:01.880041 23159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 13:19:01.880025 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 13:19:01.880115 23153 server_base.cc:1034] running on GCE node
I20260926 13:19:01.880476 23153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 13:19:01.880772 23153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 13:19:01.881933 23153 hybrid_clock.cc:648] HybridClock initialized: now 1790428741881926 us; error 42 us; skew 500 ppm
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 13:19:01.884434 23153 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 13:19:01.884827 23153 webserver.cc:460] Webserver started at http://127.0.98.60:35673/ using document root <none> and password file <none>
I20260926 13:19:01.885004 23153 fs_manager.cc:359] Metadata directory not provided
I20260926 13:19:01.885044 23153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 13:19:01.885129 23153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 13:19:01.886106 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data/instance:
uuid: "cd2a4da42a7c49868f81c34d69722744"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "8b46f42f6354176976a9917dbc9c66c6"
server_key_iv: "e43470d6a1a58238a0c580b315f4fa8b"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.886531 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal/instance:
uuid: "cd2a4da42a7c49868f81c34d69722744"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "8b46f42f6354176976a9917dbc9c66c6"
server_key_iv: "e43470d6a1a58238a0c580b315f4fa8b"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.887929 23153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 13:19:01.888602 23168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 13:19:01.888756 23153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 13:19:01.888844 23153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal
uuid: "cd2a4da42a7c49868f81c34d69722744"
format_stamp: "Formatted at 2026-09-26 13:19:01 on dist-test-slave-f864"
server_key: "8b46f42f6354176976a9917dbc9c66c6"
server_key_iv: "e43470d6a1a58238a0c580b315f4fa8b"
server_key_version: "encryptionkey@0"
I20260926 13:19:01.888899 23153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 13:19:01.892899 23146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } attempt: 2
I20260926 13:19:01.893283 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.893164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52548)
0926 13:19:01.893221 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.893257 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 13:19:01.893667 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.893590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52560)
0926 13:19:01.893629 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.893652 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 13:19:01.893766 23146 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
I20260926 13:19:01.916713 23071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } attempt: 2
I20260926 13:19:01.917241 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.917101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52564)
0926 13:19:01.917170 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.917212 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 13:19:01.917580 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.917499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52580)
0926 13:19:01.917540 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.917564 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 13:19:01.917692 23071 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37447: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37447: connect: Connection refused (error 111)
I20260926 13:19:01.927688 23153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 13:19:01.929725 23153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 13:19:01.929833 23153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 13:19:01.933823 23153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37447
I20260926 13:19:01.933873 23220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37447 every 8 connection(s)
I20260926 13:19:01.934279 23153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/data/info.pb
I20260926 13:19:01.935250 23221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 13:19:01.936708 23221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
I20260926 13:19:01.941228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskLNUYBk/build/release/bin/kudu as pid 23153
I20260926 13:19:01.941371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskLNUYBk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790428444292953-392-0/minicluster-data/master-2/wal/instance
I20260926 13:19:01.941603 392 external_mini_cluster.cc:1428] Setting key a16cde05497e3d435c83bb5796b64cec
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 13:19:01.946810 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.937468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.0.98.62:38923)
0926 13:19:01.937547 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.937549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.937711 (+ 162us) server_negotiation.cc:407] Connection header received
0926 13:19:01.938274 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.938275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.938288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.938350 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.938985 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.939474 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.940466 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.940668 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.940755 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.943415 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.943416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.943416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.943424 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.945114 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.945632 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.945634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.945634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.945685 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.945991 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.945992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.945992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.946139 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.946254 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.946608 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.946690 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 13:19:01.946844 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.937342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38923 (local address 127.0.0.1:51130)
0926 13:19:01.937574 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.937625 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.938121 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.938423 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.938427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.938684 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.938874 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.938879 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.939585 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.939586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.940364 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.940366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.940782 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.940783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.940897 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.941434 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.941453 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.943325 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.945208 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.945213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.945215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.945542 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.945760 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.945761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.945762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.945901 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.946269 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.946276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.946469 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.946563 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.946682 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
I20260926 13:19:01.947489 23221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:19:01.951390 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.942027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52596 (local address 127.0.98.60:37447)
0926 13:19:01.942209 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.942214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.942319 (+ 105us) server_negotiation.cc:407] Connection header received
0926 13:19:01.942383 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.942385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.942403 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.942484 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.943229 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.944085 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.945313 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.945790 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.945860 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.948521 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.948528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.948529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.948535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.950184 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.950556 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.950559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.950560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.950623 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.950826 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.950827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.950828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.950974 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.951146 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.951192 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.951271 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:19:01.954528 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.947745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43193 (local address 127.0.0.1:35008)
0926 13:19:01.947797 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.947807 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.947842 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.948011 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.948012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.948160 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.948228 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.948230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.949076 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.949077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.949701 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.949703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.950240 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.950241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.950305 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.950671 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.950676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.951764 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.953482 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.953484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.953484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.953662 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.953872 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.953873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.953873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.953929 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.954202 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.954202 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.954246 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.954331 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.954434 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 13:19:01.954528 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.947826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.61:43193)
0926 13:19:01.947893 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.947896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.947904 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:19:01.947935 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.947935 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.947943 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.948006 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.948314 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.948993 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.949783 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.950122 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.950246 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.951861 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.951863 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.951863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.951870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.953371 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.953738 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.953739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.953740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.953802 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.954003 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.954004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.954005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.954118 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.954214 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.954356 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.954434 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:19:01.954924 23221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:19:01.959728 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.952165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.0.98.62:38923)
0926 13:19:01.952218 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.952224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.952233 (+ 9us) server_negotiation.cc:407] Connection header received
0926 13:19:01.952256 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.952257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.952274 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.952343 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.952802 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.953249 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.953904 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.954117 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.954234 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.956638 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.956640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.956640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.956648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.958485 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.958871 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.958872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.958872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.958928 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.959171 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.959171 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.959171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.959313 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.959524 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.959569 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.959632 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 13:19:01.961817 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.955218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52608)
0926 13:19:01.955258 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.955268 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.955302 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.955407 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.955408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.955547 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.955574 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.955575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.956198 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.956199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.956688 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.956690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.957093 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.957094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.957142 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.957546 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.957550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.958936 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.960796 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.960796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.960797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.960963 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.961170 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.961171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.961171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.961224 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.961485 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.961485 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.961523 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.961609 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.961705 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 13:19:01.961835 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.955278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.0.98.60:37447)
0926 13:19:01.955314 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.955315 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.955323 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:19:01.955342 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.955343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.955353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.955397 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.955663 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.956121 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.956783 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.957002 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.957093 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.959042 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.959043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.959043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.959050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.960708 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.961046 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.961047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.961048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.961096 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.961296 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.961297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.961297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.961404 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.961490 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.961631 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.961751 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":21}
I20260926 13:19:01.963635 23221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: Bootstrap starting.
I20260926 13:19:01.964754 23221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: Neither blocks nor log segments found. Creating new log.
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 13:19:01.965090 23221 log.cc:824] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: Log is configured to *not* fsync() on all Append() calls
I20260926 13:19:01.965921 23221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: No bootstrap required, opened a new log
I20260926 13:19:01.967424 23221 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:01.967582 23221 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2a4da42a7c49868f81c34d69722744, State: Initialized, Role: FOLLOWER
I20260926 13:19:01.967680 23221 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:01.967988 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:01.968112 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:19:01.968060 23221 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:19:01.968212 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.960562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35020 (local address 127.0.98.61:43193)
0926 13:19:01.960608 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.960610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.960620 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:19:01.960660 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.960661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.960669 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.960715 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.961194 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.961663 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.962609 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.962794 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.962847 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.965361 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.965363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.965363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.965370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.966848 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.967211 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.967213 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.967214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.967270 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.967468 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.967469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.967469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.967580 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.967756 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.967968 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.968048 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 13:19:01.969709 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 13:19:01.970067 23238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:19:01.970122 23238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:19:01.970818 23045 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "17990373fe2c49b8bb3aee6b20bcd478"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51140
I20260926 13:19:01.972162 23153 master_runner.cc:186] Error getting master registration for 127.0.98.62:38923: OK, instance_id { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" instance_seqno: 1790428741572915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:19:01.974329 23153 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:19:01.984936 23146 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } attempt: 3
Sep 26 13:19:01 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:19:01.992666 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.985395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52614)
0926 13:19:01.985454 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:01.985471 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:01.985532 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:01.985683 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:01.985684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:01.985849 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:01.985893 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.985894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.986652 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.986653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.987185 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:01.987187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.987578 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.987579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:01.987646 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.988154 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:01.988159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:01.989430 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:01.991432 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.991433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.991434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.991677 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.991896 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:01.991897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:01.991897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:01.991959 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:01.992314 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:01.992315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:01.992368 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 13:19:01.992440 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 13:19:01.992554 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 13:19:01.992707 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:01.985490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.0.98.60:37447)
0926 13:19:01.985563 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:01.985565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:01.985575 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:19:01.985602 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:01.985603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:01.985615 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:01.985667 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:01.985991 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.986561 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.987287 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:01.987501 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:01.987580 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:01.989546 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:01.989547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:01.989548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:01.989557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:01.991311 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.991764 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.991766 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.991766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.991830 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:01.992027 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:01.992028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:01.992029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:01.992224 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:01.992298 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:01.992488 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 13:19:01.992573 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 13:19:01.995047 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: Bootstrap starting.
I20260926 13:19:01.996068 23146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: Neither blocks nor log segments found. Creating new log.
I20260926 13:19:01.996433 23146 log.cc:824] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: Log is configured to *not* fsync() on all Append() calls
I20260926 13:19:01.997435 23146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: No bootstrap required, opened a new log
I20260926 13:19:01.999917 23146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:02.000085 23146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93d819f0d9c14a0d92ea09aa2f4b242f, State: Initialized, Role: FOLLOWER
I20260926 13:19:02.000181 23146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:02.000510 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.000593 23146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:19:02.000625 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: This master's current role is: FOLLOWER
W20260926 13:19:02.002748 23252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:19:02.002825 23252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:19:02.003096 23078 master_runner.cc:186] Error getting master registration for 127.0.98.62:38923: OK, instance_id { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" instance_seqno: 1790428741572915 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 13:19:02.005990 23078 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:19:02.016819 23071 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } attempt: 3
Sep 26 13:19:02 dist-test-slave-f864 krb5kdc[22990](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790428741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 13:19:02.024827 23074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:02.017173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37447 (local address 127.0.0.1:52630)
0926 13:19:02.017228 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 13:19:02.017242 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 13:19:02.017292 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 13:19:02.017468 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 13:19:02.017479 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 13:19:02.017647 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 13:19:02.017691 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:02.017693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.018648 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.018650 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:02.019154 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 13:19:02.019156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.019615 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.019615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 13:19:02.019673 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:02.020167 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 13:19:02.020173 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 13:19:02.021605 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 13:19:02.023712 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:02.023713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:02.023714 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:02.023905 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:02.024110 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 13:19:02.024111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 13:19:02.024126 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 13:19:02.024186 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 13:19:02.024462 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 13:19:02.024463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 13:19:02.024509 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 13:19:02.024611 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 13:19:02.024722 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 13:19:02.024824 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:02.017311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.0.98.60:37447)
0926 13:19:02.017365 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.017367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.017375 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:19:02.017399 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.017400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.017411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.017463 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.017784 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.018579 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.019252 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.019555 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.019617 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:02.021696 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:02.021697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:02.021698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:02.021708 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:02.023637 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.023999 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.024001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.024001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.024060 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.024237 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.024238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.024239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.024386 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:02.024456 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:02.024637 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 13:19:02.024722 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:19:02.026784 23071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478: Bootstrap starting.
I20260926 13:19:02.027590 23071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478: Neither blocks nor log segments found. Creating new log.
I20260926 13:19:02.027910 23071 log.cc:824] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478: Log is configured to *not* fsync() on all Append() calls
I20260926 13:19:02.028702 23071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478: No bootstrap required, opened a new log
I20260926 13:19:02.029736 23071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:02.029896 23071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17990373fe2c49b8bb3aee6b20bcd478, State: Initialized, Role: FOLLOWER
I20260926 13:19:02.029989 23071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:02.030293 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.030406 23045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 13:19:02.030357 23071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 13:19:02.030436 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:19:02.030463 23045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:19:02.031286 23045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:02.031584 23045 leader_election.cc:288] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [CANDIDATE]: Term 1 election: Requested vote from peers 93d819f0d9c14a0d92ea09aa2f4b242f (127.0.98.61:43193), cd2a4da42a7c49868f81c34d69722744 (127.0.98.60:37447)
I20260926 13:19:02.031909 23120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17990373fe2c49b8bb3aee6b20bcd478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f"
I20260926 13:19:02.032069 23120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:19:02.032819 23195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17990373fe2c49b8bb3aee6b20bcd478" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cd2a4da42a7c49868f81c34d69722744"
I20260926 13:19:02.032960 23195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [term 0 FOLLOWER]: Advancing to term 1
I20260926 13:19:02.033126 23120 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17990373fe2c49b8bb3aee6b20bcd478 in term 1.
I20260926 13:19:02.033869 23195 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17990373fe2c49b8bb3aee6b20bcd478 in term 1.
I20260926 13:19:02.034129 23019 leader_election.cc:302] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17990373fe2c49b8bb3aee6b20bcd478, cd2a4da42a7c49868f81c34d69722744; no voters:
I20260926 13:19:02.039662 23255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 13:19:02.039930 23255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [term 1 LEADER]: Becoming Leader. State: Replica: 17990373fe2c49b8bb3aee6b20bcd478, State: Running, Role: LEADER
I20260926 13:19:02.040107 23255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } }
I20260926 13:19:02.040704 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17990373fe2c49b8bb3aee6b20bcd478. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.040866 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: This master's current role is: LEADER
W20260926 13:19:02.041564 23270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 13:19:02.041671 23270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 13:19:02.043145 23276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 13:19:02.043345 23276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 13:19:02.045158 23003 master_runner.cc:263] No leader master found from master 17990373fe2c49b8bb3aee6b20bcd478
I20260926 13:19:02.045212 23003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 13:19:02.045601 23120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [term 1 FOLLOWER]: Refusing update from remote peer 17990373fe2c49b8bb3aee6b20bcd478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:19:02.045601 23195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [term 1 FOLLOWER]: Refusing update from remote peer 17990373fe2c49b8bb3aee6b20bcd478: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 13:19:02.046061 23255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:19:02.046226 23279 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 13:19:02.048069 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17990373fe2c49b8bb3aee6b20bcd478. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.048157 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:19:02.048888 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17990373fe2c49b8bb3aee6b20bcd478. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.049013 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:19:02.049827 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.049947 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: This master's current role is: LEADER
I20260926 13:19:02.050318 23227 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.050405 23227 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:19:02.050688 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.050699 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17990373fe2c49b8bb3aee6b20bcd478" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17990373fe2c49b8bb3aee6b20bcd478" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38923 } } peers { permanent_uuid: "93d819f0d9c14a0d92ea09aa2f4b242f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43193 } } peers { permanent_uuid: "cd2a4da42a7c49868f81c34d69722744" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37447 } } }
I20260926 13:19:02.050773 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 13:19:02.050786 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478 [sys.catalog]: This master's current role is: LEADER
I20260926 13:19:02.051043 23276 catalog_manager.cc:1347] Generated new cluster ID: 3162740e7f284496bab2594e2bea6ed2
I20260926 13:19:02.051086 23276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 13:19:02.051434 23280 mvcc.cc:205] Tried to move back new op lower bound from 7333596127415201792 to 7333596127397654528. Current Snapshot: MvccSnapshot[applied={T|T < 7333596127415201792}]
I20260926 13:19:02.059132 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:02.035924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35024 (local address 127.0.98.61:43193)
0926 13:19:02.035972 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.035975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.051224 (+ 15249us) server_negotiation.cc:407] Connection header received
0926 13:19:02.051295 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.051296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.051308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.051366 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.052194 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.052919 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.053917 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.054235 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.054276 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:02.055886 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:02.055888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:02.055888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:02.055895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:02.057570 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.058269 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.058270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.058270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.058336 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.058550 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.058550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.058551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.058670 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:02.058748 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:02.058949 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 13:19:02.059033 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 13:19:02.062207 23073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:02.036433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.0.98.62:38923)
0926 13:19:02.036479 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.036481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.054333 (+ 17852us) server_negotiation.cc:407] Connection header received
0926 13:19:02.054438 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.054439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.054449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.054478 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.055260 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.055982 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.057001 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.057294 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.057349 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:02.059166 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:02.059168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:02.059168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:02.059175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:02.060773 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.061164 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.061165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.061165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.061222 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.061415 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.061416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.061416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.061542 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:02.061638 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:02.061994 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 13:19:02.062071 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 13:19:02.064069 23276 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 13:19:02.064627 23276 catalog_manager.cc:1504] Loading token signing keys...
I20260926 13:19:02.069065 23276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17990373fe2c49b8bb3aee6b20bcd478: Generated new TSK 0
I20260926 13:19:02.069267 23276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 13:19:02.070691 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:02.036623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.0.98.60:37447)
0926 13:19:02.036669 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.036671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.058138 (+ 21467us) server_negotiation.cc:407] Connection header received
0926 13:19:02.058196 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.058198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.058211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.058269 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.062828 (+ 4559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.063578 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.064627 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:02.064914 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:02.064953 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:02.066530 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 13:19:02.066531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 13:19:02.066532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 13:19:02.066538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 13:19:02.068190 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.069714 (+ 1524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.069715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.069715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.069781 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 13:19:02.070023 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 13:19:02.070024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 13:19:02.070024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 13:19:02.070197 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 13:19:02.070324 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:02.070517 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 13:19:02.070591 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 13:19:02.094862 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.093188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.0.98.61:43193)
0926 13:19:02.093232 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.093235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.094187 (+ 952us) server_negotiation.cc:407] Connection header received
0926 13:19:02.094298 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.094300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.094310 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.094342 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.094771 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35036: BlockingRecv error: recv got EOF from 127.0.0.1:35036 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 13:19:02.095628 23073 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.093616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.0.98.62:38923)
0926 13:19:02.093655 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.093657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.095113 (+ 1456us) server_negotiation.cc:407] Connection header received
0926 13:19:02.095221 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.095222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.095233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.095263 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.095564 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51160: BlockingRecv error: recv got EOF from 127.0.0.1:51160 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 13:19:02.096303 23225 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.093913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.0.98.60:37447)
0926 13:19:02.093951 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.093952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.095846 (+ 1894us) server_negotiation.cc:407] Connection header received
0926 13:19:02.095939 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.095940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.095950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.095979 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.096247 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52646: BlockingRecv error: recv got EOF from 127.0.0.1:52646 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 13:19:02.098559 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.097780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.0.98.61:43193)
0926 13:19:02.097814 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.097816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.098038 (+ 222us) server_negotiation.cc:407] Connection header received
0926 13:19:02.098139 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.098140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.098157 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.098212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.098514 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35042: BlockingRecv error: recv got EOF from 127.0.0.1:35042 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 13:19:02.101583 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.100800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.0.98.61:43193)
0926 13:19:02.100844 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.100846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.101105 (+ 259us) server_negotiation.cc:407] Connection header received
0926 13:19:02.101201 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.101202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.101212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.101241 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.101535 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35046: BlockingRecv error: recv got EOF from 127.0.0.1:35046 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 13:19:02.105455 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.104719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.98.61:43193)
0926 13:19:02.104752 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.104754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.104967 (+ 213us) server_negotiation.cc:407] Connection header received
0926 13:19:02.105059 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.105060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.105070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.105099 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.105399 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35056: BlockingRecv error: recv got EOF from 127.0.0.1:35056 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 13:19:02.111626 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.110727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.0.98.61:43193)
0926 13:19:02.110776 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.110778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.111057 (+ 279us) server_negotiation.cc:407] Connection header received
0926 13:19:02.111161 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.111163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.111185 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.111215 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.111577 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35070: BlockingRecv error: recv got EOF from 127.0.0.1:35070 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 13:19:02.122076 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.121070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.98.61:43193)
0926 13:19:02.121118 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.121120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.121416 (+ 296us) server_negotiation.cc:407] Connection header received
0926 13:19:02.121544 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.121545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.121559 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.121593 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.122011 (+ 418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35082: BlockingRecv error: recv got EOF from 127.0.0.1:35082 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 13:19:02.142387 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.141434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.0.98.61:43193)
0926 13:19:02.141492 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.141494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.141703 (+ 209us) server_negotiation.cc:407] Connection header received
0926 13:19:02.141819 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.141820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.141833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.141866 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.142312 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35096: BlockingRecv error: recv got EOF from 127.0.0.1:35096 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 13:19:02.176895 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.175923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.98.61:43193)
0926 13:19:02.175969 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.175971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.176275 (+ 304us) server_negotiation.cc:407] Connection header received
0926 13:19:02.176386 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.176387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.176399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.176430 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.176836 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35112: BlockingRecv error: recv got EOF from 127.0.0.1:35112 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 13:19:02.243789 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.243144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.98.61:43193)
0926 13:19:02.243234 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.243236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.243247 (+ 11us) server_negotiation.cc:407] Connection header received
0926 13:19:02.243324 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.243325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.243334 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.243377 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.243726 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35118: BlockingRecv error: recv got EOF from 127.0.0.1:35118 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 13:19:02.373764 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.373069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.0.98.61:43193)
0926 13:19:02.373166 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.373168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.373229 (+ 61us) server_negotiation.cc:407] Connection header received
0926 13:19:02.373271 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.373271 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.373282 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.373352 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.373713 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35132: BlockingRecv error: recv got EOF from 127.0.0.1:35132 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 13:19:02.632058 23150 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 13:19:02.631254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.0.98.61:43193)
0926 13:19:02.631363 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:02.631365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:02.631383 (+ 18us) server_negotiation.cc:407] Connection header received
0926 13:19:02.631410 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:02.631410 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:02.631421 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:02.631488 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 13:19:02.631980 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35140: BlockingRecv error: recv got EOF from 127.0.0.1:35140 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 13:19:02.970574 23238 catalog_manager.cc:1259] Loaded cluster ID: 3162740e7f284496bab2594e2bea6ed2
I20260926 13:19:02.970660 23238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: loading cluster ID for follower catalog manager: success
I20260926 13:19:02.971537 23238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: acquiring CA information for follower catalog manager: success
I20260926 13:19:02.971872 23238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd2a4da42a7c49868f81c34d69722744: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:19:03.003275 23252 catalog_manager.cc:1259] Loaded cluster ID: 3162740e7f284496bab2594e2bea6ed2
I20260926 13:19:03.003368 23252 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: loading cluster ID for follower catalog manager: success
I20260926 13:19:03.004202 23252 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: acquiring CA information for follower catalog manager: success
I20260926 13:19:03.004552 23252 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93d819f0d9c14a0d92ea09aa2f4b242f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 13:19:03.148108 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:03.145260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35154 (local address 127.0.98.61:43193)
0926 13:19:03.145387 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:03.145389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:03.145401 (+ 12us) server_negotiation.cc:407] Connection header received
0926 13:19:03.145479 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:03.145480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:03.145531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:03.145597 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:03.145740 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.146347 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.147198 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.147405 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.147456 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:03.147659 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:03.147789 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:03.147869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:03.148013 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 13:19:03.148035 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":96,"threads_started":1}
I20260926 13:19:03.152546 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:03.150185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.0.98.60:37447)
0926 13:19:03.150330 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:03.150331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:03.150341 (+ 10us) server_negotiation.cc:407] Connection header received
0926 13:19:03.150403 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:03.150404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:03.150440 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:03.150524 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:03.150606 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.151128 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.151808 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.152019 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.152078 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:03.152208 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:03.152316 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:03.152371 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:03.152460 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 13:19:03.152478 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":57,"threads_started":1}
I20260926 13:19:03.157325 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:03.154304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51162 (local address 127.0.98.62:38923)
0926 13:19:03.154419 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:03.154421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:03.154549 (+ 128us) server_negotiation.cc:407] Connection header received
0926 13:19:03.154839 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:03.154840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:03.154896 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:03.154966 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:03.155139 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.155630 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.156386 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.156612 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.156696 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:03.156885 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:03.157001 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:03.157039 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:03.157174 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 13:19:03.157207 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260926 13:19:03.157593 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:03.154609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.0.98.60:37447)
0926 13:19:03.154666 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:03.154668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:03.154676 (+ 8us) server_negotiation.cc:407] Connection header received
0926 13:19:03.154732 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:03.154733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:03.154752 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:03.154812 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:03.154927 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.155654 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.156786 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.157055 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.157130 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:03.157288 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:03.157318 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:03.157364 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:03.157489 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 13:19:03.157522 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 13:19:03.158190 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 13:19:03.154304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.0.98.61:43193)
0926 13:19:03.154346 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 13:19:03.154348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 13:19:03.154704 (+ 356us) server_negotiation.cc:407] Connection header received
0926 13:19:03.154737 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 13:19:03.154737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 13:19:03.154750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 13:19:03.154803 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 13:19:03.154976 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.155700 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.156823 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 13:19:03.157031 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 13:19:03.157116 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 13:19:03.157349 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 13:19:03.157395 (+ 46us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 13:19:03.157448 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 13:19:03.158064 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 13:19:03.158087 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 13:19:03.159305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 23003
I20260926 13:19:03.165043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 23078
I20260926 13:19:03.170710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskLNUYBk/build/release/bin/kudu with pid 23153
2026-09-26T13:19:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2890 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5693 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298883 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS